Spine measurement system and method therefor

ABSTRACT

A spine measurement system comprises an optical measurement probe, one or more targets, a fluoroscope, and a remote station. A-P and lateral images of the spine are taken using the fluoroscope and provided to the remote station. The remote station includes computer vision that can identify endplates and pedicle screws in the spine. The computer vision in the remote station is further used to identify vertebra and bone landmarks of the spine. The remote station can generate quantitative measurement data such as Cobb angles and axial rotation of the spine from the fluoroscope images that correspond to the spine deformity. The optical measurement probe can send images of the spine with pedicle screw extenders extending from the pedicle screws to the remote station. The remotes station using computer vision can provide spine metrics in real-time by tracking position of the pedicle screw extenders.

CROSS-REFERENCE TO RELATED APPLICATIONS FIELD

The present invention pertains generally to measurement of physicalparameters, and particularly to, but not exclusively, medical electronicdevices for high precision measurement of the spine.

BACKGROUND

The skeletal system of a mammal is subject to variations among species.Further changes can occur due to environmental factors, degradationthrough use, and aging. An orthopedic joint of the skeletal systemtypically comprises two or more bones that move in relation to oneanother. Movement is enabled by muscle tissue and tendons attached tothe musculoskeletal system. Ligaments can position, hold, and stabilizeone or more bones of a joint. Cartilage is a wear surface that preventsbone-to-bone contact, distributes load, and lowers friction.

There has been substantial growth in the repair of the human skeletalsystem. In general, orthopedic joints have evolved using informationfrom simulations, mechanical prototypes, and patient data that iscollected and used to initiate improved designs. Similarly, the toolsbeing used for orthopedic surgery have been refined over the years buthave not changed substantially. Thus, the basic procedure for correctionof the musculoskeletal system has been standardized to meet the generalneeds of a wide distribution of the population. Although the tools,procedure, and artificial replacement systems meet a general need, eachreplacement procedure is subject to significant variation from patientto patient. The correction of these individual variations relies on theskill of the surgeon to adapt and fit the replacement joint using theavailable tools to the specific circumstance. It would be of greatbenefit if a system could be developed that improves surgical outcomesand reduces the cost and time of a surgery.

BRIEF DESCRIPTION OF THE DRAWINGS

Various features of the system are set forth with particularity in theappended claims. The embodiments herein, can be understood by referenceto the following description, taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 illustrates a spine measurement system in accordance with anexample embodiment;

FIG. 2 is an illustration of the display of the remote station with apre-operative plan in accordance with an example embodiment;

FIG. 3 is an illustration of a lateral fluoroscope image of a lumbarregion of the spine in accordance an example embodiment;

FIG. 4 is an illustration of a lateral fluoroscope image of the lumbarregion of the spine in accordance with an example embodiment;

FIG. 5 is an illustration of the spine measurement system having theoptical measurement probe and a target coupled to the spine inaccordance to the example embodiment;

FIG. 6 is an illustration of the display of the remote station inaccordance with an example embodiment;

FIG. 7 is an illustration of the display of the remote station inaccordance with an example embodiment;

FIG. 8A is block diagram of a method of spine alignment in accordancewith an example embodiment;

FIG. 8B is a continuation of the block diagram 8A;

FIG. 9 is block diagram of the optical measurement probe in accordancewith an example embodiment;

FIG. 10 is an illustration of the optical measurement probe inaccordance with an example embodiment;

FIG. 11 illustrates component layout within the optical measurementprobe in accordance with an example embodiment;

FIG. 12 is an illustration of a sealed compartment of the opticalmeasurement probe in accordance with an example embodiment;

FIG. 13 illustrates the camera and electronic circuitry coupled togetherin accordance with an example embodiment;

FIG. 14 is an illustration of the mount in accordance with an exampleembodiment;

FIG. 15 is a partial view of the enclosure of the optical measurementprobe illustrating release features in accordance with an exampleembodiment;

FIG. 16 is a partial view of the enclosure of the optical measurementprobe illustrating release features in accordance with an exampleembodiment;

FIG. 17A is a block diagram of a method of spine measurement inaccordance with an example embodiment;

FIG. 17B is a continuation of the block diagram 17A;

FIG. 18 is an illustration of a rod coupled to a lower lumbar region inaccordance with an example embodiment;

FIG. 19 is a block diagram of a method to support pedicle screwplacement in accordance with an example embodiment;

FIG. 20 is a lateral view of the lower lumbar region illustrating adrill trajectory in accordance with an example embodiment;

FIG. 21 is a transverse view of the L3 vertebra illustrating a drilltrajectory in accordance with an example embodiment;

FIG. 22 is an illustration of the spine measurement system configured tosupport pedicle screw placement in accordance with an exampleembodiment;

FIG. 23 is an illustration of feedback to support directing the tool tothe point of entry on the targeted vertebra in accordance with anexample embodiment;

FIG. 24 depicts an exemplary diagrammatic representation of a machine inthe form of a spine measurement system in accordance with an exampleembodiment;

FIG. 25 is an illustration of a communication network for measurementand reporting in accordance with an example embodiment;

FIG. 26 is an illustration of the spine measurement system in accordancewith an example embodiment;

FIG. 27 is an illustration of the spine measurement system in accordancewith an example embodiment;

FIG. 28 is a side view image and a top view image of a rod and anencoded collar in accordance with an example embodiment;

FIG. 29 is a block diagram of a method of measuring a shape of a rod fora spine in accordance with an example embodiment;

FIG. 30 is a block diagram illustrating using computer vision toidentify or recognize an object in accordance with an exampleembodiment;

FIG. 31 is a lateral fluoroscope image of a portion of a spine showingpedicle screws in vertebrae in accordance with an example embodiment;

FIG. 32 is a binary image of FIG. 31 showing pedicle screws inaccordance with an example embodiment;

FIG. 33 is an image showing regions of interest in accordance with anexample embodiment;

FIG. 34 is a block diagram illustrating the use of computer vision forpedicle screw or vertebra identification from a fluoroscope image or animage provided by an optical measurement probe in accordance with anexample embodiment;

FIG. 35 is a block diagram illustrating a Cobb angle measurement inaccordance with an example embodiment;

FIG. 36 is a block diagram illustrating a tracking of targets such aspedicle screw extenders in accordance with an example embodiment;

FIG. 37 is a block diagram illustrating a pose derived 3D location inspace from a 2D image in accordance with an example embodiment;

FIG. 38 is block diagram illustrating a rod measurement in accordancewith an example embodiment;

FIG. 39 is block diagram of an automated orthopedic process using one ormore fluoroscope images to locate musculoskeletal structures ororthopedic devices and the position of each structure or device relativeto one another to generate quantitative measurement data in accordancewith an example embodiment;

FIG. 40 is a block diagram illustrating steps involved with a computerand computer vision software to identify musculoskeletal structures ordevices in one or more images provided to the computer in accordancewith an example embodiment;

FIG. 41 is a block diagram illustrating steps involved with a computerand computer vision software to place and identify musculoskeletalstructures or orthopedic devices in one or more images provided to thecomputer in accordance with an example embodiment;

FIG. 42A is a block diagram illustrating steps involved with a computerand computer vision software to place and identify musculoskeletalstructures or orthopedic devices in one or more images provided to thecomputer in accordance with an example embodiment;

FIG. 42B is a continuation of the block diagram in FIG. 42A inaccordance with an example embodiment;

FIG. 43 is a block diagram illustrating tracking of one or more objectsusing computer vision software in real-time in accordance with anexample embodiment;

FIG. 44 is a block diagram having further detail illustrating trackingof one or more identified objects using computer vision software inreal-time in accordance with an example embodiment;

FIG. 45 is a block diagram having further detail illustrating imagetraining using computer vision software in real-time to track one ormore objects in accordance with an example embodiment;

FIG. 46 is a block diagram showing steps to acquire a position of amusculoskeletal structure or orthopedic device in a video frame inreal-time in accordance with an example embodiment; and

FIG. 47 is an illustration of three pedicle screws where an edgedetection algorithm was applied to the image.

DETAILED DESCRIPTION

Embodiments of the invention are broadly directed to measurement ofphysical parameters, and more particularly, to a system that supportsaccurate measurement, improves surgical outcomes, reduces cost, reducestime in surgery.

The following description of exemplary embodiment(s) is merelyillustrative in nature and is in no way intended to limit the invention,its application, or uses.

Processes, techniques, apparatus, and materials as known by one ofordinary skill in the art may not be discussed in detail but areintended to be part of the enabling description where appropriate. Forexample specific computer code may not be listed for achieving each ofthe steps discussed, however one of ordinary skill would be able,without undo experimentation, to write such code given the enablingdisclosure herein. Such code is intended to fall within the scope of atleast one exemplary embodiment.

In all of the examples illustrated and discussed herein, any specificmaterials, such as temperatures, times, energies, and materialproperties for process steps or specific structure implementationsshould be interpreted to be illustrative only and non-limiting.Processes, techniques, apparatus, and materials as known by one ofordinary skill in the art may not be discussed in detail but areintended to be part of an enabling description where appropriate. Itshould also be noted that the word “coupled” used herein implies thatelements may be directly coupled together or may be coupled through oneor more intervening elements.

Additionally, the sizes of structures used in exemplary embodiments arenot limited by any discussion herein (e.g., the sizes of structures canbe macro (centimeter, meter, and larger sizes), micro (micrometer), andnanometer size and smaller).

Notice that similar reference numerals and letters refer to similaritems in the following figures, and thus once an item is defined in onefigure, it may not be discussed or further defined in the followingfigures.

In general, a prosthesis is an artificial body part. An orthopedicimplant is a device used to repair the musculoskeletal system. Commonexamples of an orthopedic implant are pins, rods, screws, cages, platesand other devices that typically couple to bone of the musculoskeletalsystem. A prosthetic joint can be part of a system that supportsmovement of the musculoskeletal system. A prosthetic joint typicallycomprises several prosthetic components that combine to mimic a naturaljoint. For example, a prosthetic hip joint comprises an acetabularshell, an acetabular bearing, a femoral prosthetic component. Theacetabular shell couples to the pelvis and is a pivot point of thejoint. The acetabular bearing fits in the acetabular shell and providesa bearing surface that supports hip movement. The femoral prostheticcomponent comprises a femoral head and a femoral hip stem. The headcouples to the hip stem and fits into the acetabular bearing todistribute loading to the bearing surface. The femoral hip step couplesto the proximal end of the femur. Thus, a prosthetic hip joint is a balland socket joint that couples the femur to the pelvis to supportmovement of the leg. Similarly, prosthetic joints are available torepair the knee, ankle, shoulder, hand, fingers, wrist, toes, and spine.

The prosthetic joint or a prosthetic component of the joint can alsohave a number of sensors for generating measurement data related to theinstallation. For example, joint position or prosthetic componentloading can be monitored in surgery or long-term. A result of themonitoring could be that an exercise regimen could be prescribed toimprove the range of motion. Similarly, balance, loading, alignment, orjoint position could be monitored or data stored to study kinematics ofthe joint or provide a kinetic assessment of the joint. Also, the jointcould be monitored for wear or pending failure. In all cases, themeasurement data can be used to enhance performance, reliability, andidentify potential failure in a time frame when it can be repaired witha minimally invasive procedure.

FIG. 1 is an illustration of a spine measurement system 10 in accordancewith an example embodiment. Spine measurement system 10 comprises anoptical measurement probe 12, one or more targets 14, mounts 16, afluoroscope 18, a drill 11 and a remote station 20. Spine measurementsystem 10 is configured to provide quantitative measurement data relatedto the spine. In one embodiment, spine measurement system 10 isconfigured for use in a surgical environment such as an OR (OperatingRoom) within a hospital to provide quantitative measurement data on theposition of vertebrae, vertebra identification, measurement of anglesrelated to vertebra, vertebral modeling, range of motion, extractkinematics (TKA, THA), and spine simulation. Corrections made to thespine can be monitored in real-time to determine a total change to aspine region of interest and if further modifications are need toachieve a desired outcome.

In general, hospital, clinics, and medical offices have reduced budgetsfor capital expenditures. At the same time, medical outcomes bothshort-term and long-term need to be improved to lower cost. Spinesurgery is especially problematic in that much of the outcome isdetermined by the skill of the surgeon. Many surgeons do not spendsufficient time in the operating room on spine surgeries to feelcomfortable with many of the required skills such as placing pediclescrews in the vertebra. The surgeon often compensates by usingtechniques that require iterative steps to ensure correct location ofthe pedicle screw. Similarly, surgeons new to spine surgery requiresubstantial time under the guidance of a skilled surgeon to train anddevelop the techniques that yield successful outcomes. Spine measurementsystem 10 improves outcomes by supporting spine surgeries withquantitative measurement data related to spine shape, pedicle screwplacement, prosthetic component placement, load measurement, and rodshape. For example, spine measurement system 10 can support pediclescrew installation by identifying the location where the pedicle screwis placed on the vertebra and in real-time time provide data to supportdrill 11 placement at a correct point of entry on a targeted vertebraand a correct trajectory of a drill path into the targeted vertebra.Spine measurement system 10 generates quantitative measurement datarelated to spine shape and measure an outcome based on real-time spinemeasurements. Furthermore, spine measurement system 10 can display thespine in real-time, generate angles of relevance to the spine, analyzethe spine based on the measurements, provide corrections, and one ormore workflows to achieve the desired outcome. The use of quantitativemeasurement data and workflows backed by clinical evidence can improvethe surgical outcome and reduce the surgical time.

As mentioned previously, capital cost of equipment can be a barrier toproviding devices that can significantly improve spine surgery outcomes.Spine measurement system 10 is designed to be low cost where some of thecomponents are disposed of after a single use. Fluoroscope 18 is part ofspine measurement system 10. Fluoroscope 18 is a common device that ispresent in the operating room for spine surgery. Thus, fluoroscope 18 isnot required for purchase thereby substantially reducing the systemcost. As used today, the fluoroscope is not used to generate anyquantitative measurements but is used to provide images of the spineduring surgery for review and verification. Alternatively, a catscan(CT) or magnetic resonance imaging can be used in place of a fluoroscopeimage. The CT or MRI would be provided to remote station 20. Opticalmeasurement probe 12, mount 16, and target 14 are disposable componentsthat can be built at low cost while providing the performance, accuracy,and reliability required to provide measurement data to support a spinesurgery. The cost of optical measurement probe 12, mount 16, and target14 can be incorporated into the cost of surgery and invoiced at the timeof surgery which eliminates an equipment capital cost or maintaining aninventory of components. Remote station 20 processes informationreceived from optical measurement probe 12 to generate quantitativemeasurement data to support the spine surgery. In one embodiment,optical measurement probe 12 comprises a camera and provides image datato remote station 20. In one embodiment, measurement system 10 cancomprise more than one camera. Remote station 20 can be purchased,leased, or given to the entity using spine measurement system 10. Nocapital expenditure is required by leasing or providing remote station20 at no cost. Providing remote station 20 at no cost can beaccommodated if surgical volumes are sufficient. Leasing of remotestation 20 may be practical for an entity having low volume of spinesurgeries while achieving better spine outcomes. In general, providingthe low cost solution requiring little or no capital expenditure and nopaid inventory is a solution to get equipment in the operating roomwhere it can benefit the surgeon, the patient, and the hospital byreducing surgical time, increasing the accuracy of the surgery, generatequantitative measurement data, reduce rehabilitation time, and improvepatient outcomes long-term.

In general, a surgeon first meets with the patient and generates adiagnosis. Each diagnosis is unique to the individual and situation. Thediagnosis may require surgery to resolve the problem. Typically, thesurgeon generates a pre-operative plan that defines the spine region ofinterest and the objectives to be achieved. The pre-operative plan isdesigned to be imported to spine measurement system 10. In oneembodiment, spine measurement system 10 can include a workflowcorresponding to the type of surgery being performed where questions areanswered by the surgeon that relate to the surgery. For example, system10 can be used to support the installation and positioning of orthopedicimplants such as pins, rods, screws, cages, plates and other devicesthat typically couple to bone of the musculoskeletal system. Spinemeasurement system 10 in the operating room can couple to the cloud anddownload the pre-operative plan. In one embodiment, all data coming tospine measurement system 10 is encrypted. Similarly, any outflow of dataor information from spine measurement system 10 is encrypted to preventothers from viewing the data. Information displayed on the spinemeasurement system 10 may include patient information, the diagnosis,the vertebrae being operated on, metrics, the hardware being used in thesurgery, goals and expected results of the surgery, a workflow of thesurgery, measurement data, analysis, and other miscellaneousinformation. In the example, spine measurement system 10 can display animage of the spine as the surgeon envisions an end result when thesurgery is completed. The pre-operative image can be compared inreal-time to the spine in surgery to determine differences between thepre-operative plan and the actual surgery. Spine measurement system 10supports changes or modifications during surgery that yield the desiredsurgical outcome or modifications due to unforeseen issues that were notseen or disclosed in the pre-operative planning.

Fixed references are used to generate a coordinate system withmeasurement probe 12. In one embodiment, spine measurement system 10 isreferenced to static objects that are in the operating room. Forexample, vents, light fixtures, switches, and other objects that do notchange position can be used as references. Typically, three staticpoints are used as reference points. Spine measurement system 10 cantake into account position changes in the spine by also identifyingreference points of the patient, spine or other patient points ofreference that correspond to the coordinate system generated by spinemeasurement system 10 using the static objects. In one embodiment,optical measurement probe 12 is used to reference the three staticpoints. Images of the references are sent to remote station 20. Remotestation 20 can view the images and determine if a position has changedand compensate future measurements for the new orientation.Alternatively, system 10 can measure the relative position of objects inthe field of view of optical measurement probe 12 whereby themeasurements are independent of the camera coordinate system.

The surgeon resects tissue, ligaments, and muscle in a spine region toexpose a portion of the spine. Fluoroscope 18 can be rotated to takeimages of the spine from different angles. Typically, anterior-posteriorimage and a lateral image are taken of the spine region of interest.Spine surgery to repair a lumbar region of the spine is used as anexample of spine measurement system 10 generating quantitativemeasurement data in support of the operation. System 10 can be used onall regions of the spine such as cervical, thoracic, lumbar, and sacralspine regions. The repair of the lumbar region in the example willcomprise fastening a rod to pedicle screws inserted in L2-L4 vertebrae.The rod is bent by the surgeon to modify the curvature of the lumbarregion of the spine when coupled to the pedicle screws. The L2-L4vertebrae are fused together to hold the new shape. The rod willmaintain the desired shape of the spine while the fusing takes hold andstrengthens to a point where the rod can be removed. It should be notedsystem 10 can be used for spine surgeries of two or more vertebraeincluding modifying the entire spine and is not limited by the example.

Fluoroscope 18 generates anterior-posterior and lateral images of eachvertebra to support placement of pedicle screws. Placement of pediclescrews can be a time consuming procedure for surgeons. The pedicle screwplacement process is iterative whereby a hole is drilled partially intothe bone and fluoroscope images are taken. A wire can also be used toprobe into the vertebra instead of drilling. The depth of the hole canbe increased after verification of a correct drill path by fluoroscopeimages. Typically, the drill path is approximately centered within thepedicle to maximize the bone area around the screw. The drill path isdifferent for each vertebra. After drilling the pilot path, the openingcan be widened and tapped to accept a pedicle screw. In one embodiment,optical measurement probe 12 can be used to take images of a display 25of fluoroscope 18 during the pedicle placement process and after thepedicle screw is placed. Optical measurement probe 12 includes a handle24 that fits in the hand allowing it to be aimed at display 26 offluoroscope 18. Optical measurement probe 12 is coupled to and sends theimages to remote station 20. In one embodiment, the surgeon can identifythe vertebra or vertebrae corresponding to the fluoroscope images.Remote station 20 uses computer vision to identify the vertebra orvertebrae from the fluoroscope images and generates a representation ofthe spine in the fluoroscope images on display 22 of remote station 20.The surgeon responds to verify that that the identification is correct.For example, the spine image can be displayed on display 22 of remotestation 20 with a box around a vertebra with a label indicating thevertebra (e.g. L1, L2, L3 . . . ). The surgeon can use switch 26 onoptical measurement probe 12 to interact with the user interface onremote station 20 to verify that the label is correct. Alternatively, ifthe vertebra label is incorrect the surgeon can use switch 26 to changethe label to the appropriate vertebra. Verification sets the vertebralsequence on display 22 since the sequence of vertebra is known.Alternatively, voice recognition can be used for the surgeon in averification process with remote station 20.

Remote station 20 processes the images from fluoroscope 18 to generatequantitative measurement data relevant to the spine surgery. Thefluoroscope images will show detail of the spine including bone detail,landmarks, pedicle screws within a vertebra and endplates of eachvertebra. Instead of taking an image of a display 25 with opticalmeasurement probe 12, fluoroscope 18 can directly send image data toremote station 20. Fluoroscopes in a hospital, clinic, or office are canvary significantly and may not port easily to remote station 20. Inother words, fluoroscope 18 would have to be set up to interface withspine measurement system 10. Images can be coupled from fluoroscope 18or optical measurement probe 12 by wired or wireless connection.

Quantitative measurement data is generated by remote station 20 usingthe images provided by fluoroscope 18. In one embodiment, thefluoroscope images are digital images. Remote station 20 comprisesmicroprocessors, digital signal processors, microcontrollers, interfacecircuitry, control logic, memory, digital circuitry, sensors, analogcircuitry, transceiver circuitry, converters, display 22, and othercircuitry. Remote station 20 can run software and can interface withdevices that interact with the external environment. In one embodiment,remote station 20 is a computer, tablet, or a portable device. Remotestation 20 can also provide feedback such as visual, audible, and hapticfeedback to a surgical team. Remote station 20 also couples to theinternet, one or more databases, and the cloud. A software programimplementing computer vision is used by remote station 20 to generatequantitative measurement data such as Cobb angle or the compression ordistraction of the distance between vertebral endplates used by surgeonsto assess the spine. For example, system 10 can get a baseline of thejunctional endplates to define predictive kyphosis or support othersimilar measurements. Computer vision attempts to recreate the abilityof human vision to perceive and understand what an image is. Computervision does not just process the image data but uses visual cues thatare common to other similar objects ascertain what is being viewed. Oneadvanced area using computer vision software is in facial recognition.For example, a computer can be programmed to recognize a fork. Not allforks look the same but have many features in common with each other.Machine vision can recognize a fork having substantial equal dimension'sto what is stored in memory. The machine vision may not recognize thefork if it is altered. Conversely, a human can view a fork that he orshe has never seen before, process the image, and determine that it is afork even with the differences. A computer with computer vision tries tomimic this human process. The computer vision software will haveidentifiers or visual cues to look for that in combination can allow theprogram to conclude that what is being shown is a fork. Moreover, theentire fork may not be visible but with sufficient identifiers locatedon the image the computer vision could conclude that it is a fork evenwith only a partial view.

Remote station 20 is configured to use computer vision software torecognize the musculoskeletal system. In one embodiment, the computervision software is configured to recognize the spine, vertebrae, andbone landmarks of each vertebra. Furthermore, the computer visionsoftware is configured to recognize equipment, tools, and componentsused in the surgery. For example, equipment such as pedicle screws andscrew extenders placed in the spine can be recognized by the computervision software. The recognition of the spine and equipment is used togenerate quantitative measurement data that is used by the surgeon inreal-time. In one embodiment, Cobb angles can be measured usingrecognition of vertebral end plates and pedicle screws. A Cobb angle isa measurement to characterize spine curvature in a region of the spine.The Cobb angle can be measured in the coronal plane usinganteroposterior fluoroscope images to indicate deformity from the ideal.Similarly, the Cobb angle can be measured using lateral fluoroscopeimages to characterize deformities or curvature in the sagittal plane.Rotational aspects of vertebra to a reference can also be measured byremote station 20. In general, measurements are not limited to thesagittal plane. Sagittal images can be combined with lateral andanterior-posterior images to utilize coronal and axial planes usinganatomical and instrument landmarks.

FIG. 2 is an illustration of display 22 of remote station 20 with apre-operative plan in accordance with an example embodiment. Display 22of remote station 20 displays a pre-operative plan for the spinesurgery. As mentioned previously, remote station 20 can be coupledthrough the internet or cloud to retrieve the information that thesurgeon has prepared. Remote station 20 will decrypt the informationreceived. The surgeon will use this information during the course of thesurgery and can refer back to it if required. In one embodiment, anelectronic form can be filled out by the surgeon related to the surgerywhereby basic information is recorded, stored, received by remotestation 20, and displayed on display 22. Moreover, specific informationor notes needed by the surgeon can be added or highlighted. Display 22can also include figures, pictures, spine scans that relate to thepre-operative plan. For example, a figure of a post-operative outcome ofthe spine region of interest based on the pre-operative plan can bedisplayed in section 35 of display 22 of remote station 20. In oneembodiment, the projected post-operative outcome could be compared tothe spine in surgery in real-time.

Display 22 can be divided into sections with pre-operative informationrelated to the surgery. A section 32 comprises patient information.Patient information displayed on display 22 of remote station 20 insection 32 can comprise patient name, patient identification number,height, body mass index, and blood type. The patient information shownillustrates what can be put in section 32 but is not limited to thisdata. Other information can be added or removed depending on what isrelevant for the specific surgery.

A section 33 comprises medical information. Medical informationdisplayed on display 22 of remote station 20 can comprise a hospital,surgeon name, medical diagnosis, a surgical date, and an operating roomnumber. The medical information shown illustrates what can be put insection 33 but is not limited to this data. Other information can beadded or removed depending on what is relevant for the specific surgery.

A section 34 comprises a pre-operative plan for the surgery.Pre-operative plan information displayed on display 22 of remote station20 can comprise pre-operative measurement information, spinemodification information, and expected post-operative outcomeinformation. The pre-operative plan information comprises a section ofthe spine to be operated on, vertebrae of interest, Cobb angles, andaxial rotation. In the example, pre-operative sagittal Cobb angle,pre-operative coronal Cobb angle, and pre-operative axial rotation areprovided on display 22. The type and amount of pre-operative informationwill vary with the type of surgery being performed. In the example, apost-operative sagittal Cobb angle, post-operative coronal Cobb angle,and a post-operative axial rotation are provided. Pre-operative andpost-operative axial rotation includes the direction of rotation. Thetype and amount of post-operative information will vary with the type ofsurgery being performed. Other information can be added or removeddepending on what is relevant for the specific surgery.

A section 36 comprises discectomy information on display 22 of remotestation 20. In the example, the surgery is being performed in the lowerlumbar region. More specifically, surgery is being performed onvertebrae L2-L4 where a spine deformity is being corrected requiringcorrection in the sagittal and coronal planes. Discectomy informationrelates to the disc material being removed from the spine. In theexample, L2-L3 and L3-L4 are identified as regions for discectomy.Further information can also be provided such as the discectomy ofvertebrae L2-L3 and L3-L4 are partial discectomies. Other informationcan be added or removed depending on what is relevant for the specificsurgery.

A section 38 comprises osteotomy information on display 22 of remotestation 20. As mentioned, the surgery is being formed in the lowerlumbar region in the example. Each vertebra of the spine surgery islisted. Osteotomy information discloses bone cuts or bone modificationsto reduce medical problems related to the spine and to support change tothe spine shape. In the example, the L3 vertebra inferior requiresmodification as disclosed in section 38. Other information can be addedor removed depending on what is relevant for the specific surgery.

A section 39 comprises instrumentation used during the surgery. Asshown, components that couple to the spine to modify spine shape arelisted. In the example, pedicle screws, a rod, and an implant are listedon display 22 of remote station 20. Each pedicle screw may have adifferent length or profile. Pedicle screws are listed corresponding toeach vertebra with information related to size and length of the screw.The rod length and diameter of the rod to modify spine shape isdisclosed. Implants used in the operation are also listed on display 22.Other information can be added or removed depending on what is relevantfor the specific information. For example, pedicle screw extenders thatcan be recognized by optical measurement probe 12 of FIG. 1 could beadded to section 39 as will be discussed hereinbelow.

FIG. 3 is an illustration of a lateral fluoroscope image 40 of a lumbarregion of the spine in accordance of an example embodiment. Lateralfluoroscope image 40 has been received by remote station 20 of FIG. 1.The lumbar region comprises L1 vertebra 42, L2 vertebra 44, L3 vertebra46, L4 vertebra 48, L5 vertebra 50, and S1 sacrum 52. The vertebrae ofthe lumbar region have been labeled in remote station 20 and verified bythe surgeon. Computer vision within remote station 20 of FIG. 1 isconfigured to identify the endplates of each vertebra. In the example,vertebrae L2-L4 are being corrected for a curvature and rotationaldeformity. The Cobb angle of the vertebrae L2-L4 is calculated fromfluoroscope image 40 and the quantitative measurement is displayed ondisplay 22 of remote station 20 of FIG. 1. The Cobb angle is used by thesurgeon as a measure of the deformity of the spine in the region ofinterest and can be compared against the pre-operative plan to determineif changes are required.

Remote station 20 of FIG. 1 can calculate Cobb angle using at least twomethods. A first method extends the planes of the endplate surfaces ofinterest to intersection and measures the angle between the planes. Inthe example, a plane of the proximal endplate of vertebra L2corresponding to the proximal endplate surface is extended from thespine. The proximal endplate plane of vertebra L2 is indicated as line54. Similarly, a plane of the distal endplate of vertebra L4corresponding to the distal endplate surface is extended from the spine.The distal endplate plane of vertebra L4 is indicated as line 56. In thefirst method, lines 55 and 56 are extended until intersection. Remotestation 20 of FIG. 1 calculates an angle between lines 55 and 56 whichis the Cobb angle 60 for the example embodiment.

Alternatively, a line 62 can be extended at a right angle from line 54.A line 64 can be extended at a right angle from line 56. In the example,both lines 62 and 64 extend interior to Cobb angle 60 from the examplehereinabove. Lines 62 and 64 are extended to intersection. Remotestation 20 of FIG. 1 calculates an angle between lines 62 and 64. Theangle is indicated by Cobb angle 58. Cobb angle 58 is equal to Cobbangle 60.

FIG. 4 is an illustration of a lateral fluoroscope image 70 of thelumbar region of the spine in accordance with an example embodiment.Referring briefly to FIG. 3, the fluoroscope image 40 did not show thepedicle screws to simplify the drawing and illustrate measurement of theCobb angle by remote station 20 of FIG. 1. Referring back to FIG. 4,lateral fluoroscope image 70 shows pedicle screws inserted duringsurgery. Fluoroscope image 70 is sent directly to remote station 20 ofFIG. 1 or optical measurement probe 12 can take an image of display 25of fluoroscope 18 of FIG. 1. As mentioned previously, computer visionwithin remote station 20 of FIG. 1 can recognize anatomy of themusculoskeletal system, components, devices, and equipment. For example,remote station 20 of FIG. 1 can recognize pedicle screws that have beeninserted in the spine. In general, remote station 20 of FIG. 1, willmeasure more than just the Cobb angle from the fluoroscope imagesalthough only the Cobb angle may be displayed of the quantitativemeasurement data. A table of the different measurements is generated andstored within memory of remote station 20 of FIG. 1 that can be recalledand used when required. For example, endplate to endplate, pedicle screwto pedicle screw, and pedicle screw to endplate measurements can bemeasured and stored in a table corresponding to the vertebrae ofinterest from the fluoroscope images.

In the example measurement, endplate to pedicle screw quantitativemeasurements are generated. Lateral fluoroscope image 70 of the spinehas been imported to remote station 20 of FIG. 1. Fluoroscope image 70includes the lower lumbar region of the spine. The lumbar regioncomprises L1 vertebra 72, L2 vertebra 74, L3 vertebra 76, L4 vertebra78, L5 vertebra 80, and S1 sacrum 82. The vertebrae of the lumbar regionis displayed and labeled on display 22 of remote station 20 of FIG. 1.The surgeon has verified that the spine image displayed on display 22 ofremote station 20 of FIG. 1 is correct.

Fluoroscope image 70 further includes a pedicle screw 84, a pediclescrew 86, and a pedicle screw 88 respectively coupled to L2 vertebra 74,L3 vertebra 76, and L4 vertebra 78. The remote station application usingcomputer vision is configured to identify the endplates of each vertebraand the pedicle screws. In the example, vertebrae L2-L4 are beingcorrected for a curvature and rotational deformity. Screw to endplateangle measurements can be used to determine an amount of correctionrequired for the spine. As mentioned, the Cobb angle is an indication ofthe amount of deformity in the spine. Pedicle screw to endplate angles,endplate to endplate angles, and pedicle screw to pedicle screw anglesquantitative measurements can be used to determine changes to individualvertebra or groups of vertebra in relation to the Cobb angle. Similarly,the same measurements can be made using an anteroposterior fluoroscopeimage of the spine for correction in the coronal plane. Also, rotationalmeasurements and correction of the spine can be identified from thelateral and anteroposterior fluoroscope images.

An angle 94 relates to a proximal endplate of L2 vertebra 72 and pediclescrew 84. Remote station 20 of FIG. 1 locates the proximal endplate ofL2 vertebra 72 and extends the plane of the proximal endplate of L2vertebra 72. The plane corresponds to a surface of the proximal endplateand is indicated by line 90. Pedicle screw 84 is coupled to L2 vertebra72. Remote station 20 of FIG. 1 locates a center of pedicle screw 84 andextends a trajectory of pedicle screw 84 to intersection with the planeof the proximal endplate of L2 vertebra 72 indicated by line 90. Thetrajectory of pedicle screw 84 is indicated by line 92. Remote station20 of FIG. 1 measures an angle 94 that is formed between lines 90 and92.

An angle 100 relates to a distal endplate of L4 vertebra 78 and pediclescrew 88. Remote station 20 of FIG. 1 locates the distal endplate of L4vertebra 78 and extends the plane of the distal endplate from L4vertebra 78. The plane corresponds to a surface of the distal endplateand is indicated by line 98. Pedicle screw 88 is coupled to L4 vertebra78. Remote station 20 of FIG. 1 locates a center of pedicle screw 88 andextends a trajectory of pedicle screw 88 to intersection with the planeof the distal endplate of L4 vertebra 78 indicated by line 98. Thetrajectory of pedicle screw 88 is indicated by line 96. Remote station20 of FIG. 1 measures an angle 100 that is formed between lines 96 and98. In general, remote station 20 using computer vision generatesquantitative measurement data comprising Cobb angles, vertebra endplateto endplate, pedicle screw to vertebra endplate, pedicle screw topedicle screw, and other information from fluoroscope images generatedduring a pedicle screw procedure. The quantitative measurement data ofthe spine can be compared to the pre-operative plan and be used in thespine modification. In general, the fluoroscope images are used togenerate quantitative measurement data that is used to characterize thespine an initial state. The initial state corresponds to the spine withthe deformity being corrected. In one embodiment, the quantitativemeasurement data from the fluoroscope images is used to create a spineimage with deformity that is displayed on display 22 of remote station20 of FIG. 1. As will be disclosed hereinbelow, the spine is monitoredduring surgery with quantitative measurement data such as Cobb anglebeing provide in real-time as the spine shape is modified. The real-timespine monitoring includes quantitative measurement of spine shape thatis interpolated with the initial spine shape to provide an image of thespine shape as it is manipulated on display 22 of remote station 20 ofFIG. 1 in real-time.

FIG. 5 is an illustration of spine measurement system 10 having opticalmeasurement probe 12 and a target 14 coupled to the spine in accordanceto the example embodiment. Optical measurement probe 12 and one or moretargets are used to measure and monitor spine position in real-timeduring a surgery. Optical measurement probe 12 includes a cameraconfigured to monitor one or more targets. In one embodiment, the camerais monocular. Alternatively, the camera can be binocular. A monocularcamera is lower cost than a binocular camera which supports a disposablequantitative spine measurement device for the operating room. Opticalmeasurement probe 12 can couple to the spine or to a reference position.Optical measurement probe 12 can also be mounted to a stable non-movingstructure such as an operating table having a view of the spine duringsurgery. In general, spine measurement system 10 is configured to takeabsolute 6 degrees of freedom measurements throughout the procedure. Inone embodiment, spine measurement system 10 is configured to use thecamera center as its coordinate system origin. In another embodiment,system 10 is configured to measure the relative position of an objectwith respect to another object in the Field Of View. One or morealgorithms can be used to identify and count the target devices orobjects in the field of view.

As previously mentioned, optical measurement probe 12 can generate areference image from the reference position. In one embodiment, threestatic objects within the within the field of view of the camera areused as references from the reference position. The optical measurementprobe 12 can be moved from the reference position and then returned at alater time. It is possible that the angle of the camera with respect tothe references can change slightly when the camera is repositioned. Thechange in angle will have subsequent affect on quantitative measurementsmade by spine measurement system 10. The camera upon being returned tothe reference position will identify the references. Remote station 20will not produce any change to image data received if the references arefound to be in the same positions. Remote station 20 will calculate athree-dimensional compensation if the references are found to bepositioned differently due to the camera being in a new position. Imagedata received by remote station 20 from optical measurement probe 12will have this offset to ensure all data is measured identically andcorresponds to the camera being in the original reference position.

As shown in the illustration, optical measurement probe 12 is coupled toa pedicle screw 28. A mount 16 couples between optical measurement probe12 and pedicle screw 28. Mount 16 can have a predetermined shape thatpositions optical measurement probe 12 from being directly over thespine. In one embodiment, mount 16 couples to handle 24 of opticalmeasurement probe 12. Handle 24 includes a quick release that allowsoptical measurement probe 12 to be removed from mount 16. This featurecan be used is to take an image of the spine or an image of display 26of fluoroscope 18 where optical measurement probe 12 can be held in handand directed at an object. Optical measurement probe 12 can bereattached to mount 16 after the images have been taken thereby placingthe device in the position it was previously in. Mount 16 is configuredto couple to a pedicle screw. In one embodiment, mount 16 screws into ahead region of pedicle screw 28. Tightening the distal end coupling ofmount 16 to pedicle screw 28 fixes a position of optical measurementprobe 12 in relation to the spine.

A camera within optical measurement probe 12 is focused on target 14.Similar to optical measurement probe 12, target 14 is coupled to apedicle screw 29. A mount 17 couples between target 14 and pedicle screw29. Mount 17 can have a predetermined shape that positions opticalmeasurement probe 12 away from the spine. This affords the surgeon moreroom to perform spine manipulation and the surgery. Target 14 can bedisengaged from mount 17 or can be permanently affixed to mount 17. Inone embodiment, mount 17 screws into a head region of pedicle screw 29.Tightening the distal end coupling of mount 16 to pedicle screw 29 fixesa position of target 14 in relation to the spine. As previouslymentioned, target 14 is in a field of view of the camera within opticalmeasurement probe 12. More than one target can be coupled to the spine.Each target is in the field of view of the camera when coupled to thespine. Mounts can be adjusted to move targets such that at least aportion of the images on each target are in the field of view.

In one embodiment, target 14 has a surface 25 with multiple images thatcan be viewed by optical measurement probe 12. In one embodiment, imageson target 14 are two-dimensional images. Alternatively, target 14 canhave three-dimensional images formed on surface 25. As shown, target 14comprises images of circles each having a different size. Opticalmeasurement probe 12 tracks movement of the spine by comparing changeson the image of target 14 to the initial image of target 14 viewed byremote station 20. In the example, optical measurement probe 12 andtarget 14 is set up to track changes of one of the L2-L4 vertebrae.Other vertebra can be tracked by adding targets. Typically, the image oftarget 14 sent by optical measurement probe 12 to remote station 20corresponds to the L2-L4 vertebrae being in the pre-operative statehaving the deformity that was measured on using the fluoroscope images.The spine can then be manipulated which changes the spine shape. Changesin the image viewed by the camera in optical measurement probe 20 can beconverted to a three dimensional movement of the vertebra to whichtarget 14 couples. The detected change in movement or rotation of thetarget is translated or rotated to a position change in 3D space of thevertebra to which the target couples. Software in remote station 20 canprocess the image data from optical measurement probe 20 and translateand rotate it to a change in spine shape and reflect this change to theimage of the spine provided on display 22.

There is a direct correlation between the position in 3D space ofoptical measurement probe 12 to the vertebra to which it couples on thespine. Similarly, there is a direct correlation between the position oftarget 14 to the vertebra to which it couples on the spine. Thedimensions of optical measurement probe 12, target 14, mount 16, mount17, pedicle screw 28, and pedicle screw 29 are known and stored inremote station 20. Remote station 20 knows the location and trajectoryof pedicle screw 28 in the vertebra to which it couples and the locationand trajectory of pedicle screw 29 in the vertebra to which it couples.Moreover, remote station 20 has stored angle values such as Cobb angle,endplate to endplate, pedicle screw to endplate, pedicle screw topedicle screw of the pre-operative spine. All of this is used to producean accurate image on display 22 of remote station 20 of the spine regionof interest in real-time.

Typically, spine surgery results in a change in spine shape. In theexample, a rod will be coupled to the pedicle screws in L2-L4. The rodis bent to adjust the curve of the spine in the sagittal plane and thecoronal plane. The rod shape can be bent to rotate vertebra. Movement ofthe spine in the region of interest will result in movement of opticalmeasurement probe 12, target 14, or both. Optical measurement probe 12has to refocus on target 14 and note differences in the image to theinitial image corresponding to the initial spine shape with deformity.Remote station 20 utilizes computer vision, real-time image data fromoptical measurement probe 12, quantitative measurement data generatedfrom the fluoroscope images, and device dimensions and informationrelated to the system components to translate the movement to changes inspine position that can be viewed on display 22 of remote station 20 inreal-time. For example, the surgeon can view changes in the spine shapeon display 22 as the spine is manipulated. Not only can the spine shapebe viewed, but quantitative measurement data is generated related to thespine shape on display 22 that comprises information such as Cobb angleor vertebral rotation. The quantitative measurement data in conjunctionwith the surgeon subjective feel related to the spine manipulation canresult in changes to the pre-operative plan. A new plan or workflow canthen be implemented by the surgeon or by remote station 20. In oneembodiment, the new plan is implemented and quantitative measurementsare taken to verify the results of the changes.

FIG. 6 is an illustration of display 22 of remote station 20 inaccordance with an example embodiment. Referring briefly to FIG. 5, thepedicle screws have been placed in the region of interest in the spine.Remote station 20, optical measurement probe 12, and target 14 arecoupled to the pedicle screws to provide quantitative measurement datarelated to the position of vertebra relative to one another. The spinedeformity is replicated by remote station 20 from the fluoroscope imagesand shown on display 22. The spine shape is generated in real-time fromquantitative measurement data calculated by remote station 20 usingcomputer vision. Remote station 20 receiving image data from opticalmeasurement probe 12 is configured to calculate the position of eachvertebra in relation to others from the targets coupled vertebrae. Thespine shape shown in display 22 changes as the spine is manipulated fromits initial position in real-time.

Referring back to FIG. 6, a table 110 is shown on display 22 of remotestation 20. The table includes a column of pre-operative data and acolumn of quantitative measurement data. The spine is manipulated by thesurgeon. The manipulation is a subjective analysis by the surgeon todetermine if the pre-operative plan is practical and if there are anyunforeseen issues. In one embodiment, the column of quantitativemeasurement data corresponds to an outcome for the spine. The spine ismanipulated to a shape that the surgeon considers a good outcome for thepatient. In the example, the lower lumbar region is of interest. Morespecifically, the shape of the L2-L4 region of the spine will bemodified with a rod coupled to the pedicle screws placed in the L2-L4vertebrae. As mentioned the spine shape is being measured in real-time.The surgeon can have remote station 20 store a spine shape that he hasmanipulated the spine into. The stored spine shape will includemeasurements of all the angles and rotation required to replicate thespine back into this position. In the example, the intra-operativecolumn on table 110 are measurements related to manipulated spine shapethat the surgeon believes yields a good outcome.

The pre-operative data of the spine can be compared to the quantitativemeasurement data generated in the operating room. Both the pre-operativedata and the initial intra-operative quantitative measurement datarelates to the spine deformity to be corrected. There can be differencesbetween what the surgeon calculated using scan information of the spineand what is measured during surgery. The difference can be due to howthe patient was measured or to unforeseen patient disease or severity ofdisease. For example, the intra-operative measurement data is generatedwith the patient lying prone on an operating table after ligamentreleases and facet osteotomies are made while the pre-operative data canbe taken with the patient in an upright position. The quantitativemeasurement data can be used to make adjustments to the pre-operativeplan to achieve the desired outcome. Moreover, the quantitativemeasurement data can provide information that was not included in thepre-operative plan.

Table 110 provides L2-L4 sagittal Cobb angle, coronal Cobb angle, andaxial rotation data. More or less information can be provided on table110 depending on the spine deformity and the type of surgery beingperformed. In the example, the intra-operative quantitative measurementsdiffer from the pre-operative measurements. For example, theintra-operative sagittal Cobb angle measures 4 degrees less than thepre-operative measurement. Similarly, the coronal Cobb angle measures 5degrees less than the pre-operative measurement. The axial rotationmeasured the same for the pre-operative and intra-operativemeasurements. The surgeon can use this information to make adjustmentsto the plan and also to redefine the outcome of the spine surgery.

Images 112 of the pre-operative spine can be provided with table 110.Similarly, images 114 of the intra-operative spine measured by spinemeasurement system 10 can be provided with table 110. Ananterioposterior view and a lateral view of the pre-operative spine andintra-operative spine are shown. Images 112 and 114 can be overlayed oneach other to highlight differences in the pre-operative spine model andthe intra-operative spine model. Alternatively, the pre-operative spinerod and the reduced spine rod could be overlayed on each other tohighlight differences. Other features to highlight the spine deformitycan be used to allow the surgeon at a glance to understand theinformation in table 110 as it relates to images 112 and 114.

FIG. 7 is an illustration of display 22 of remote station 20 inaccordance with an example embodiment. As discussed previously, thesurgeon can manipulate the spine to determine if there are issues withimplementing the pre-operative plan or intra-operative plan. Computervision software in remote station 20 identifies a position of eachtarget and translates the target position to a position of acorresponding vertebra. The changes in position are calculated anddisplayed on display 22. The changes can be displayed in real-time asthe spine is manipulated. Typically, the real-time changes will besupported with a quantitative measurement such as sagittal Cobb angle,coronal Cobb angle, and axial rotation. In one embodiment, the surgeoncan provide a modified plan based on the intra-operative measurementsduring surgery. Remote station 20 can provide a surgical workflow andwith changes from the pre-operative plan that achieve the desiredoutcome. For example, remote station 20 can calculate appropriate bendsin a rod that couples to pedicle screws to affect a spine shapecorresponding to the modified plan.

In the example, a rod has been coupled to the spine. The rod is clampedto the pedicle screws with the one or more targets in place. The rodwill have one or more bends that affect vertebral placement in thesagittal and coronal planes. Thus, the spine shape takes a shape of therod. Computer vision software in remote station 20 measures the spineshape with the rod and targets in place. An anterioposterior view and alateral view of the reduced spine shape in the L2-L4 region of interestis displayed on the display 22 as images 116. Images 112 correspondingto the pre-operative spine outcome are also shown for comparison. Acolumn is added to table 110 that shows the quantitative measurement ofthe spine having been “reduced” with the insertion and clamping of therod to the pedicle screws. As shown, the measured values differ from thepre-operative data and the intra-operative measured data. In particular,the reduced spine shape has a sagittal Cobb angle of 25 degrees, acoronal Cobb angle of 7 (right) degrees, and an axial rotation of 5degree (clockwise). The surgeon can review the outcome with the rod inplace and determine if the changes meet the desired outcome or ifchanges need to be made. Remote station 20 can also analyze the outcomeand suggest changes to the rod with calculated results related tosagittal Cobb angle, coronal Cobb angle, and axial rotation. In oneembodiment, the curvature of the pre-operative plan, intra-operativemeasurements, and the reduced measurements could be overlayed ordisplayed separately to show differences in each stage of the operation.

FIG. 8 is block diagram 119 of a method of spine alignment in accordancewith an example embodiment. A method of spine alignment using a spinemeasurement system is disclosed. The method can be practiced with moreor less steps and is not limited to the order of steps shown. The methodis not limited to the spine example but can be used for hip, knee,shoulder, ankle, elbow, spine, hand, wrist, foot, bone, andmusculoskeletal system. The components listed in the method can bereferred to and are disclosed in FIG. 5. In a step 120, the system isstarted. As part of the start up process a spine measurement system isturned on. The spine measurement system comprises a remote station,fluoroscope, optical measurement probe, and one or more targets. A logo,application name, and version of the software is displayed on a displayof the remote station. In one embodiment, the remote station can beplaced outside the sterile zone of an operating room but in a locationwhere the display of the remote station is visible to a surgical team.

In general, the remote station is configured to receive image data fromthe optical measurement probe. The optical measurement probe includes acamera. The camera can be monocular or binocular. The opticalmeasurement probe is configured to view a position of one or moretargets within a field of view of the optical measurement probe. Theremote station includes a remote station application that can processimage data from the optical measurement probe that generatesquantitative measurement data related to spine shape and metrics usedfor spine surgery. For example, the remote station application canmeasure a Cobb angle from fluoroscope images of the spine with theremote station application using computer vision.

In a step 122, case information is provided. Patient information can beentered directly into the remote station. Alternatively, patientinformation can be imported from local storage or through the internetat a remote location. A pre-operative plan for the surgical procedure isentered to the remote station. The pre-operative plan can be importedfrom local storage or through the internet at a remote location. Also,levels involved in the case are identified. The information can bedisplayed on the display of the remote station.

In a step 124, the optical measurement probe is used to take an image ofa display of the fluoroscope showing a fluoroscope image of the spine.The fluoroscope image can show the pedicle screws placed in thevertebrae. The optical measurement probe can be held in hand to aim thecamera at the display of the fluoroscope to take one or more images. Theimages of the fluoroscope display are sent from the optical measurementprobe to the remote station.

The remote station application using computer vision can identifyvertebra endplates and pedicle screws from the fluoroscope images. Theremote station application can be configured to interpolate a planetrajectory of a vertebra endplate or interpolate a pedicle screwtrajectory. An angle is measured between two interpolated trajectoriesthat would extend to intersection. The angle can relate to a measure ofspine deformity or spine curvature. In a first embodiment, the remotestation application is configured to calculate an angle formed byinterpolated plane trajectories of two different vertebra endplates thatintersect from the fluoroscope images. In a second embodiment, theremote station application is configured to calculate an angle formed bya plane trajectory of a vertebra endplate and a trajectory of a pediclescrew that intersect from fluoroscope images. In a third embodiment, theremote station application is configured to calculate an angle betweentwo pedicle screw trajectories that intersect from two pedicle screws intwo different vertebra from fluoroscope images. In a fourth embodiment,a Cobb angle is displayed on the display of the remote station. The Cobbangle is a measure of an angle between the plane trajectory of aproximal endplate of a proximal vertebra of the spine region of interestand the plane trajectory of the distal endplate of a distal vertebra ofthe spine region of interest.

The remote station application using computer vision locates features onthe fluoroscope images and generates metrics or quantitative measurementdata that is related to spine shape. A spine image is created from themetrics and displayed on the display of the remote station. In oneembodiment, human interaction is required for verification that thespine image on the display of the remote station is correct. The surgeoncan confirm that labeled vertebra of the spine displayed on the displayof the remote station correlates to what the surgeon views on theoperating room table.

In a step 126, pedicle screw extender's used for minimally invasivespine surgery (MISS) is performed. The pedicle screws have been placedin the spine. The pedicle screw extenders are coupled to pedicle screwsin the spine. The pedicle screw extenders extend from the spine and arevisible to the surgeon. In one embodiment, the spine, pedicle screws,and pedicle screw extenders are in the field of view of the opticalmeasurement probe.

The optical measurement probe can be coupled to a pedicle screw or astable surface. One or more targets can be coupled to a pedicle screwusing a mount having a predetermined shape. In one embodiment, thepedicle screw extenders are targets of the spine measurement system. Aview of the pedicle screws or spine is often obstructed by the hands ofthe surgeon, tools, or devices coupled to the spine. The pedicle screwextenders allow the surgeon to see a spine shape while manipulating thespine.

The remote station is configured to receive image data from the opticalmeasurement probe when coupled to the spine. The remote station isconfigured to measure the relative position of each target within thefield of view of the optical measurement probe. In one embodiment, thecamera in the optical measurement probe views the pedicle screwextenders extending from the spine. The remote station receives imagesof the pedicle screw extenders in real-time and uses computer vision toidentify each individual pedicle screw extender. Moreover, the remotestation application uses computer vision to extrapolate positions ofeach pedicle screw extender back to the vertebra to which it couplesthereby relating pedicle screw extender position to vertebra position.The remote station application measures and calculates vertebra positionas the spine is manipulated by converting changes in the 2-dimensionalimage of the pedicle screw extenders viewed by the optical measurementprobe into 3-dimension movement of the vertebra coupled to the pediclescrew extenders in real-time. The quantitative measurements are used tochange the spine shape in real-time on the display of the remote stationfrom the initial spine shape measured from the fluoroscope images.

The pedicle screw extender surfaces can have one or more images that canbe recognized by computer vision software. In one embodiment, the remotestation application will assume that the most distal pedicle screwextender in the field of view is also the most distal vertebra in thespine region of interest. The remote station can then correlate eachvertebra in the spine region of interest from the most distal vertebraand generate an image with each vertebra labeled on the display of theremote station. The user can also redefine the orientation where themost distal vertebra is the closest vertebra in the field of view. Theremote station has fluoroscope data related to pedicle screw position,placement angles, and trajectory and can use this information to supportthe real-time spine measurement. Similarly, the remote station hasdimensions of the pedicle screws and pedicle screw extenders that can beused to extrapolate back to a corresponding vertebra.

In a step 128, a rod template is provided to support shaping a rod. Theshape of the spine is measured by the remote station application anddisplayed on the display of the remote station. The surgeon manipulatesthe spine to determine a spine shape that meets a desired outcome. Theremote station application can calculate metrics and angles related tothe spine shape based on the position of the pedicle screw extenders.Thus, the spine shape as it is being manipulated by the surgeon isdisplayed on the display of the remote station in real-time. In oneembodiment, the surgeon can lock-in an in-situ spine shape. For example,the surgeon is manipulating the spine and viewing the spine shape on thedisplay of the remote station with real-time measurements. Oncelocked-in, the remote station application can calculate metrics andangles related to the in-situ spine shape such as Cobb angle. The remotestation can provide a rod template that achieves the in-situ spine shapegenerated by the surgeon.

In a step 130, a shape of the pre-operative plan rod shape is comparedto an intra-operative rod shape determined by quantitative measurement.The quantitative measurement data generated by the remote station can beused by the surgeon to determine the intra-operative rod shape. In oneembodiment, the remote station application can calculate rod angles androd shape to achieve the locked-in spine shape. In another embodiment,the remote station application can display a 1:1 image of a rod thatachieves the desired outcome. The surgeon can bend the rod to the rodtemplate displayed on the display of the remote station. Alternatively,a machine can bend the rod using quantitative measurement data providedby the remote station. The pre-operative rod shape can be ghosted oroverlayed with the intra-operative rod shape to view differences in eachshape with metrics.

The surgeon has a choice of choosing between the different rod shapes.The pre-operative rod shape and intra-operative rod shape can also beapplied to the measured intra-operative spine shape in the remotestation. The remote station can simulate the spine shape and display thespine shape with either the pre-operative rod shape, the intra-operativerod shape, or both. One or more metrics such as Cobb angle or vertebrarotation can be calculated by the remote station using the pre-operativeor intra-operative rod shape and displayed on the remote station. Theremote station application can take into account material properties ofthe rod and the force applied to the rod by the musculoskeletal systemto change rod shape to achieve the desired outcome.

In a step 132, the rod is reduced by the surgeon. The surgeon couplesthe rod to the pedicle screws. The remote station application cancalculate Cobb angles from the positions of the pedicle screw extenderswith the rod in place. The rod can be modified further if the measuredCobb angles do not achieve the desired outcome.

In a step 134, the information related to spine the surgery can beuploaded. The remote station uploads designated case data to a securedatabase. The user will exit the remote station and end the session.

FIG. 9 is block diagram of optical measurement probe 12 in accordancewith an example embodiment. Optical measurement probe 12 and remotestation 20 are disclosed in FIG. 1 as components of spine measurementsystem 10. Optical measurement probe 12 is a high-resolution camerasystem configured to monitor the musculoskeletal system and morespecifically the spine during surgery. The camera can be monocular.Optical measurement probe 12 is coupled to a remote station usingcomputer vision to generate quantitative measurement data to supportspine surgery. In one embodiment, optical measurement probe 12 is used asingle time and disposed of after a surgery has been completed.

An auto-focus lens 140 couples to an image sensor 142. Image sensor 142couples to auto focus control IC 144 to provide input that is used tocontrol auto-focus lens 140. Auto-focus lens 140, image sensor 142, andauto focus control IC 144 form a feedback path that provides a focusedimage to image sensor 142 that is within a field of view of opticalmeasurement probe 12. Image sensor 142 receives light through auto-focuslens 140 and converts the light to a digital representation of the lightpattern received. Image sensor 142 couples the digital image data to USBcontroller 146. USB controller 146 is an interface for coupling imagedata to remote station 10. Remote station 10 can then analyze the imagedata using the remote station application and computer vision softwareto generate quantitative measurement data.

Switch 26 couples to USB controller 146. Switch 26 couples to a generalpurpose input/output (GPIO) of USB controller 146. In one embodiment,switch 26 is a three-position switch for navigating a user interface onremote station 20. USB controller 146 couples a signal from the switchto remote station 20. This allows the surgeon in the sterile field ofthe operating room to navigate and make selections on a display ofremote station 20. In one embodiment, switch 26 on optical measurementprobe 12 can perform up, down, and select functions. USB controller 146can sense a switch closure from switch 26 and send a code to remotestation 20. Remote station 20 can then interpret the code and move orselect the active field in the display to complete the command.

FIG. 10 is an illustration of optical measurement probe 12 in accordancewith an example embodiment. Optical measurement probe 12 includes acamera for providing images of a spine to support generation ofquantitative measurement data used during surgery. The camera furthersupports measurement of spine alignment and relative position ofvertebra in real-time during surgery. In one embodiment, opticalmeasurement probe 12 is used for a single surgery and disposed of in anappropriate manner after the surgery has been completed.

An enclosure 160 of optical measurement probe 12 comprises a moldedstructure 156 and a molded structure 158. Molded structures 156 and 158comprise a polymer material that includes structural ribbing to increasestrength of enclosure 160 when coupled together. In one embodiment,molded structures comprise polycarbonate that can reliably andrepeatedly be formed to meet standards required in the medical devicefield. Molded structures 156 and 158 are coupled together to form ahousing for the camera and electronic circuitry. The electroniccircuitry is coupled to the camera within the housing. In oneembodiment, the electronic circuitry supports focus control, imaging,data transfer, and system control. In one embodiment, enclosure 160includes a first compartment that houses the electronic circuitry andcamera and a second compartment that couples to mount 16. The firstcompartment is sealed to prevent solids, liquids, and gases fromentering or leaving the first compartment. The second compartment doesnot have to be sealed. Optical measurement probe 12 is sterilized andplaced in sterile packaging to prevent contamination before use.

A lens cover 150 couples to enclosure 160. Lens cover 150 is sealed tomolded structures 156 and 158. In one embodiment, lens cover 150 istransmissive and distortion free to light and acts as a barrier toprotect an auto-focus lens of the camera from an external environment.Lens cover 150 can have refractive qualities and work in conjunctionwith the auto-focus lens to provide better imaging.

Switch 26 couples to enclosure 160. Switch 26 is sealed to moldedstructures 156 and 158. In one embodiment, switch 26 is a rocker switchhaving three positions. Switch 26 is used to control a user interface ona remote station while the surgeon is in a surgical field. For example,switch 26 can allow the user to move up, move down, or select fieldsshown on a display of the remote station. For example, the userinterface can be used to verify that the computer vision software on theremote station identified a vertebra or vertebrae of the spinecorrectly. In one embodiment, image data or control signals from opticalmeasurement probe 12 are coupled to the remote station via a wiredconnection. Cable 154 couples to enclosure 160. Cable 154 is sealed tomolded structures 156 and 158. For example, a grommet can be used toform a seal between cable 154 and enclosure 160. Silicone sealant couldbe used to further seal an interface between the cable, grommet, andenclosure. Alternatively, the electronic circuitry can include atransceiver. Optical measurement probe 12 could then transmit image dataand control signals wirelessly to the remote station.

Optical measurement probe 12 includes handle 24. Handle 24 allows a userto hold optical measurement probe 12 in a hand and direct the device totake images. In one embodiment, optical measurement probe 12 is held bythe surgeon to take an image of a display of a fluoroscope. The displayof the fluoroscope has a fluoroscope image that is captured by opticalmeasurement probe 12. The fluoroscope image is provided to the remotestation to generate quantitative measurement data that is used duringthe surgery. In a second embodiment, optical measurement probe 12 can becoupled to the musculoskeletal system or to a stable surface of theoperating room. For example, mount 16 can couple optical measurementprobe 12 to a pedicle screw placed in a vertebra. Enclosure 160 includesa release 152 to decouple mount 16 from optical measurement probe 12.Depressing release 152 releases mount 16 from a retaining featureallowing optical measurement probe to be removed from mount 16.

FIG. 11 illustrates component layout within optical measurement probe 12in accordance with an example embodiment. Optical measurement probe 12comprises lens cover 150, camera 162, a printed circuit board 168, aprinted circuit board 170, switch 26, and cable 154. In general,electronic circuitry is coupled to camera 162 to control and transmitimage data. Electronic components are mounted on printed circuit board168 and printed circuit board 170. Printed circuit boards 168 and 170include interconnect to couple the mounted electronic components to formcircuits.

Camera 162 comprises auto-focus lens 140, image sensor 142, and aconnector 164. The components of camera 162 can be mounted on a printedcircuit board. Connector 164 of camera 162 couples to a correspondingconnector on printed circuit board 168. Connector 164 places camera 162in a predetermined position relative to opening 166. In one embodiment,printed circuit board 168 comprises auto-focus control integratedcircuit of FIG. 9.

Molded structure 156 forms a compartment 184 and a compartment 186 whencoupled to molded structure 158 of FIG. 10. Camera 162 and theelectronic circuitry are housed in compartment 184 and sealed from theexternal environment. The electronic circuitry comprises printed circuitboards 168 and 170. Printed circuit boards 168 and 170 can includeauto-focus control integrated circuit 144 and USB controller 146 of FIG.9.

A glue channel 182 is formed on a surface of molded structure 156 thatmates with a corresponding surface of molded structure 158 of FIG. 10.In one embodiment, glue channel 182 is a groove formed on the surface ofmolded structure 156. The groove can hold glue while coupling moldedstructure 156 to molded structure 158 of FIG. 10. Referring briefly toFIG. 12, a cross-sectional view of molded structures 156 and 158 areillustrated coupled together. A compartment 184 is sealed to isolate theelectronic circuitry and camera 162 from an external environment. Thecorresponding surface of molded structure 158 can include a tongue 188that fits in glue channel 182. Tongue 188 provides more surface area forthe glue to adhere to and supports sealing molded structures 156 and 158from the external environment.

Referring back to FIG. 11, compartment 186 is not sealed from anexternal environment. In one embodiment, compartment 186 is a handlethat when held in hand can be used to direct optical measurement probe12. Compartment 186 has an opening to receive mount 16. Mount 16 couplesoptical measurement probe 12 to the muscular-skeletal system or anothersurface. In one embodiment, mount 16 couples optical measurement probe12 to a pedicle screw. Compartment 186 includes one or more features toretain and release mount 16.

In one embodiment, electronic circuitry and camera 162 are coupled tomolded structure 156. Lens cover 150 is placed in opening 164 of moldedstructure 156. One or more retention structures can hold lens cover 150in a predetermined position and seal lens cover 150 to molded structure156. In one embodiment, a cynacrolate adhesive is applied to lens cover150 and molded structure 156. The cynacrolate will seal and attach lenscover 150 to molded structure 156 within opening 164. Retaining features174 support and retain printed circuit board 168 to molded structure156. Referring briefly to FIG. 13, retaining features 174 retain printedcircuit board 168 and camera 162 in a fixed position relative to lenscover 150. In one embodiment, retaining features 174 align auto-focuslens 140 and image sensor 142 to lens cover 150 by placing printedcircuit board 168 in a predetermined position. Printed circuit board 168is slotted between retaining features 174. In general, retainingfeatures prevent movement in the vertical and horizontal directions.Retaining features 174 can flex allowing an interference fit to preventmovement. Retaining features 174 can include a locking mechanism thatoverlies an edge of printed circuit board 168 when a distal edge ofprinted circuit board 168 is adjacent to an interior surface of moldedstructure 156. Printed circuit board 168 can be released by bendingretaining features 174 away the proximal edge and removing printedcircuit board 168.

Referring back to FIG. 11, printed circuit board 170 is mounted tomolded structure 156. Printed circuit board 170 is mounted at a90-degree angle to printed circuit board 168. A flexible interconnect172 couples the electronic circuitry of printed circuit board 168 to theelectronic circuitry of printed circuit board 170. In particular,flexible interconnect couples to a connector 178 on printed circuitboard 168 to a connector 176 on printed circuit board 170. Referringbriefly to FIG. 13, molded structure 156 includes standoffs 190 thatlocate printed circuit board 170 in a predetermined position. Printedcircuit board 170 can be coupled to standoffs 190 by screws or otherfastening means. Structural ribbing 192 is formed in conjunction withstandoffs 190. In one embodiment, structural ribbing 192 comprisesstandoffs 190 and walls formed perpendicular to an interior surface ofmolded structure 156. Structural ribbing 192 increases the structuralintegrity of optical measurement probe 12.

Referring back to FIG. 11, printed circuit board 168 couples to printedcircuit board 170 through flexible interconnect 178 to support thetransfer of image data from camera 162 to a remote station located adistance from optical measurement probe 12. In one embodiment, a USBcontroller 146 of FIG. 9 is included on printed circuit board 170 tosupport transfer of the image data from camera 162 and control signalsfrom switch 26. In general, the predetermined position of printedcircuit board 170 corresponds to a position of switch 26 and cable 154.Switch 26 aligns to and couples to printed circuit board 170. In oneembodiment, switch 26 couples to actuators mounted on printed circuitboard 170 for providing control signals. Cable 154 terminates in aconnector 180 that couples to a connector mounted on printed circuitboard 170. Cable 154 couples to and aligns with printed circuit board170 to affix the connector of cable 154 to the connector on printedcircuit board 170. One or more retaining features prevent the connectorsfrom disengaging and allows cable 154 to directly exit opticalmeasurement probe 12. Furthermore, the one or more retaining featurescan include a strain relief to hold cable 154 in place. The distal endof cable 154 also terminates in a connector. Cable 154 has sufficientlength to couple optical measurement probe 12 to the remote station toprovide control signals and image data. Alternatively, printed circuitboard 170 can include a transceiver to wirelessly transmit controlsignals, transmit image data, or receive information.

FIG. 14 is an illustration of mount 16 in accordance with an exampleembodiment. Mount 16 comprises a proximal end coupling 202, a shaft 204,and a distal end coupling 206. In general, mount 16 mounts a device topatient anatomy or a stable surface within an operating room. Mount 16maintains a known geometric or spatial relationship between the deviceand the object to which it couples. In one embodiment, the relationshipbetween the device and object is fixed and does not change throughoutthe surgery. Moreover, the dimensions and shape of mount 16 is known andprovided to remote station 20. In the example, mount 16 may be usedinterchangeably with mount 17 of FIG. 5 as both are used to couple adevice such as an optical measurement probe or target to the spine or asurface and each operates similarly. Referring briefly to FIG. 5,pedicle screw 29 is coupled to a vertebra of the spine. Mount 17 is usedto couple target 14 to pedicle screw 29. The positional relationshipbetween target 14 and pedicle screw 29 is fixed by the predeterminedshape and dimensions of mount 17. Similarly, the positional relationshipbetween optical measurement probe 12 and pedicle screw 28 is also fixedby the predetermined shape and dimensions of mount 16. Also, informationrelated to the pedicle screw dimensions and pedicle screw placementwithin the vertebra is known and stored in remote station 20. Thus, amovement of target 14 can be extrapolated to movement of the vertebra towhich it is coupled using the remote station application and computervision.

Referring back to FIG. 14, proximal end coupling comprises a crossbar214 and a crossbar 216. Crossbars 214 and 216 couple to a device ortarget. In one embodiment, retaining features couple to shaft 204,crossbar 214, and crossbar 216 to rigidly hold the device to mount 16.The retaining features can be releasable whereby the mount 16 can bedecoupled from the retaining features. Shaft 204 has a predeterminedshape. In one embodiment, the shape of shaft 204 supports placement ofthe device away from where distal end coupling 206 attaches.Furthermore, shaft 204 can have one or more bends to support locatingthe device in a predetermined position. As mentioned distal end coupling206 couples to patient anatomy or a fixed surface. In the example,distal end coupling 206 couples to a pedicle screw. Distal end couplingcomprises a sleeve 218 that overlies shaft 204. A thumbwheel 210 couplesto sleeve 218 to provide a finger grip to rotate sleeve 218. Sleeve 218further includes a threaded portion 212. In one embodiment, threadedportion 212 is on an exterior of sleeve 218. A base 208 terminates adistal end of shaft 204. Base 208 of mount 16 fits into an internalfeature of the pedicle screw and creates a locked, collinearconstruction. In one embodiment, base 208 is cylindrical in shape. Base208 has a larger diameter than shaft 204. Base 208 and crossbar 216retains sleeve 218 on shaft 204.

In the example, mount 16 couples to a pedicle screw. A pedicle screw isdesigned to retain and hold a rod to that forcibly changes a contour ofa spine. The head of the pedicle screw is built up having a region toretain the rod to the pedicle screw. Typically, the head of the pediclescrew includes an interior threaded cavity. The rod is placed in thehead and a set screw is coupled to the interior threaded cavity to clampdown on the rod. Distal end coupling 206 couples to the interiorthreaded cavity of a head of a pedicle screw. Distal end coupling 206can be used to hold down a rod to the head of the pedicle screw orfasten to the pedicle screw without the rod in place. For example,threaded portion 212 engages with the interior threaded cavity of thepedicle screw. Thumb wheel 210 is rotated until base 208 couples to asurface of the pedicle screw. Further tightening of distal end coupling206 applies a force from sleeve 218 to base 208 and the surface of thepedicle screw. Thus, sleeve 218 clamps base 208 to the surface of thepedicle screw. Distal end coupling 206 can be fastened to the pediclescrew whereby shaft 206 cannot move or rotate. Thus, the device coupledto proximal end coupling 202 is fixed in place to the pedicle screw.Movement of the vertebra and pedicle screw results in a correspondingmovement of the device.

FIG. 15 is a partial view of molded structure 156 and molded structure158 illustrating release features in accordance with an exampleembodiment. The partial view shows components that form handle 24 ofFIG. 10. Handle 24 of FIG. 10 allows optical measurement probe 12 ofFIG. 10 to be held in hand and directed to take one or more images.Mount 16 is retained in compartment 186 of handle 24. Compartment 186includes a portion of opening 232 for receiving mount 16 into handle 24and retaining features for holding and releasing mount 16.

The partial view shows an exterior view of handle 24 of molded structure156 and an interior view of handle 24 of molded structure 158. Theexterior view of handle 24 of molded structure 156 includes a slot 224and a slot 226. Slot 224 and slot 226 are patterned openings formedthrough molded structure 156. Slot 224 and slot 226 are patterned inhandle 24 to form a release mechanism 152 that when pressed supportsmovement of mount 16. Slot 224 and slot 226 form a flexing region inhandle 24 that moves inward when pressed. Release mechanism 152 includesa raised region that can be easily pressed by a thumb or finger.

The interior view of handle 24 of molded structure 158 comprisesretaining features 226, cradle 234, and a portion of opening 232.Opening 232 is formed when molded structures 156 and 158 are coupledtogether. Mount 16 is inserted through opening 232. Cradle 234 has acurved surface for receiving mount 16. Cradle 234 guides mount 16 intocompartment 186 when inserted through opening 232. Retaining features226 are formed on either side of cradle 234. Retaining features 226provide a spring force to retain mount 16 when mount 16 is inserted intocompartment 186. Curved surfaces 228 couple to mount 16 and are retainedby a force applied by retaining features 226 applied to curved surfaces228 on mount 16.

FIG. 16 is a partial view of molded structure 156 and molded structure158 illustrating release features in accordance with an exampleembodiment. The partial view shows an exterior view of handle 24 ofmolded structure 158 and an interior view of molded structure 156. Theexterior view of molded structure 158 includes patterned slots 230formed through handle 24. Slots 230 are patterned to allow movement ofretaining features 226. In one embodiment, retaining features 226 aremolded having a curved lever arm that extends into compartment 186 toprovide the spring force to retain mount 16.

The interior view of handle 24 of molded structure 156 includes cradle220, posts 222, and a portion of opening 232. Mount 16 is insertedthrough opening 232. Cradle 220 has a curved surface for receiving mount16. Cradle 220 guides mount 16 into compartment 186 when insertedthrough opening 232. Mount 16 is prevented from going further than astop 236 in compartment 186. Mount 16 includes a crossbar 214 and acrossbar 216 to support retention of mount 16 in handle 24. Referringbriefly to FIG. 15, inserting mount 16 will engage crossbar 214 toretaining features 226. In particular, curved surfaces 228 will engagewith the curved surface of crossbar 214 to hold mount 16 in place.Retaining features can also engage crossbar 216. As disclosed, retainingfeatures 226 have a spring force that forcibly engages curved surfaces228 to the curved surfaces of crossbar 214 whereby mount 16 is held inplace.

Referring back to FIG. 16, mount 16 can be released from handle 24.Posts 222 align with crossbar 214. Posts 214 couple to release mechanism152. Pressing release mechanism 152 pushes posts 222 into crossbar 214moving mount 16 away from cradle 220. As mount 16 moves away from cradle220 retaining features 226 are moved towards cradle 234. Alternatively,posts 222 can push into surfaces on features 226 above cradle 220. Inthis embodiment, cradle 220 expands retaining crossbar 214 and releasingmount 16. As noted, retaining features 226 are flexible having slots 230to support movement. Mount 16 can be removed when curved surfaces 228disengage from curved surfaces of crossbar 214. Retaining features 226continue to move towards cradle 234 until curved surfaces 228 fall belowcradle 234 wherein crossbar 214 is no longer engaged by retainingfeatures 226. Mount 16 can then be pulled out through opening 232 torelease optical measurement probe 12. Target 14 of FIG. 5 can alsoinclude retaining and release features to couple to mount 17 asdisclosed herein.

FIG. 17 is a block diagram 250 of a method of spine measurement inaccordance with an example embodiment. The method of spine measurementcan use spine measurement system 10 as disclosed in FIG. 1 and FIG. 5.The method can be practiced with more or less steps and is not limitedto the order of steps shown. The method is not limited to the spineexample but can be used for hip, knee, shoulder, ankle, elbow, spine,hand, wrist, foot, bone, and musculoskeletal system. Referring brieflyto FIG. 1, a model of the spine is created from measurements generatedin a remote station application using computer vision on remote station20. A spine image is displayed on display 22 of remote station 20. Thespine image is created from quantitative measurements made by the remotestation from fluoroscope images of the spine. Referring back to FIG. 17,in a step 252, a fluoroscope is used to take lateral images andanterioposterior images of the spine. The fluoroscope images are sent toa remote station. The remote station measures one or more angles fromthe fluoroscope images. The remote station can also measure vertebraaxial rotation. A spine image is displayed on a display of the remotestation. The spine image comprises the angle measurements and axialrotation made from the fluoroscope images. In one embodiment, thefluoroscope images depict the spine with deformities (pre-correction).

In a step 254, endplates of one or more vertebra are identified from thefluoroscope images. The remote station application uses computer visionto locate the endplates of each vertebra. In general, computer visionsoftware attempts to mimic how humans identify objects. Facialrecognition software is one form of computer vision software. Facialrecognition software can often identify a face even if the face has beenaltered to obscure features. Examples of altering looks are growingfacial hair, coloring hair, changing skin tone, or significantlychanging a hairstyle. Similarly, obscuring features can include wearinga scarf, hat, or glasses to block a portion of the face. A human wouldtry to recognize features that are unique to the individual even withalteration or obfuscation. Computer vision in the remote stationapplication can be used to identify features, objects, or landmarksrelated to the musculoskeletal system or musculoskeletal surgery. In theexample, patient musculoskeletal systems can vary significantly but eachwill have a similar number of vertebrae and each vertebra will havedifferent shape that can be recognized as the features or landmarks canbe viewed in light of the differences or variations. The computer visionsoftware is programmed to recognize each component of the spine evenwith variations seen across the population. The remote stationapplication using computer vision will identify bone landmarks or otherunique features to identify a specific vertebra or a specific feature ona vertebra. As mentioned previously, the vertebrae in the fluoroscopeimages have been identified. A verification process can also beperformed whereby the surgeon verifies that the depiction of the spineon the display of the remote station corresponds to what is seen on theoperating table. The endplates of each vertebra have also beenidentified by the remote station application. Endplates are identifiedfor angle measurement either pre-operatively as part of a pre-operativeplan or during the operation by selecting endplates for measurement. TheCobb angle can be measured in sagittal or coronal planes for theselected endplates. The plane trajectory of the selected endplates isinterpolated. In general, the plane trajectory of the selected endplatesshould intersect. The angle between the interpolated plane trajectoriesof selected endplates are measured. The measured angle is the Cobb anglewhich is used by the surgeon as a relative measure of the deformity ofthe spine. The remote station can be configured to measure all therelevant endplate to endplate angles and store them in a table. Theselected endplate angle or Cobb angle can be retrieved from the table.

In a step 256, pedicle screws in the vertebra from the fluoroscopeimages are identified. The remote station application using computervision is configured to locate pedicle screws in the spine. Pediclescrews are selected for angle measurement. Pedicle screws are identifiedfor angle measurement either pre-operatively as part of a pre-operativeplan or during the operation by selecting pedicle screws for measuring.A trajectory of the selected pedicle screws is interpolated. In oneembodiment, the remote station is configured to interpolate the pediclescrew trajectories of the selected pedicle screws. The remote station isfurther configured to measure an angle between the interpolated pediclescrew trajectories of the selected pedicle screws. Alternatively, theremote station can measure all the relevant pedicle screw to pediclescrew angles and store them in a table. The selected pedicle screw topedicle screw angle can be retrieved from the table.

In a step 258, endplates of one or more vertebra are identified from thefluoroscope images as disclosed hereinabove. Similarly, pedicle screwsare identified from the fluoroscope images as disclosed hereinabove. Theremote station application using computer vision is configured to locatethe endplates of the vertebra and the pedicle screws. An endplate isselected and a pedicle screw is selected. A trajectory of the selectedendplate is interpolated. The remote station is configured tointerpolate the plane trajectory of the endplate. The trajectory of thepedicle screw is interpolated. The remote station is configured tointerpolate the pedicle screw trajectory. An angle is measured betweenthe plane trajectory of the selected endplate and the trajectory of thepedicle screw. Alternatively, the remote station can measure allrelevant pedicle screw to endplate angles and store them in a table. Theselected pedicle screw and the selected endplate can be retrieved fromthe table.

In a step 260 a camera is mounted. The spine is in a field of view ofthe camera. In one embodiment, the camera is mounted to the spine.Alternatively, the camera can be mounted to a surface that is stable andis not subject to movement. One or more targets are coupled to thespine. The targets are in the field of view of the camera. The camera isconfigured to send image data to the remote station. The remote stationusing computer vision is configured to measure positions of eachvertebra from the position of the targets. In general, position changein a target can be interpolated back to a change in position of thevertebra to which it couples. Thus, the shape of the spine is knownthrough the position of the targets. How each vertebra is positioned toone another can be represented by one or more metrics, angles, or axialrotation. The measured positions of the vertebrae in real-time can berelated or integrated with the spine image created from the fluoroscopeimages of the spine deformity. As mentioned previously, these measuredmetrics are used to generate the spine image displayed on the display ofthe remote station in real-time. A spine or spine rod image is displayedon a display of the remote station.

In a step 262 a pedicle screw extenders are coupled to pedicle screws onthe spine. The pedicle screw extenders are configured to be targets forthe remote station application. The remote station application usingcomputer vision can recognize pedicle screw extenders as targets. Thepedicle screw extenders are coupled to the spine to allow the surgeon tosee the spine shape while manipulating the spine. The pedicle screwextenders can have distinctive features that support recognition of bythe remote station application in a surgical environment.

In a step 264 the spine is manipulated. A surgeon manipulates the spineto determine if the pre-operative plan can be implemented, the limitsthat the spine can be shaped, and determine if any unknown issues exist.The manipulation is a subjective analysis of the spine and can includeshaping the spine to an acceptable outcome. The remote stationapplication measures the spine shape in real-time during the spinemanipulation. Real-time metrics are displayed on the display of theremote station corresponding to the spine shape and other attributes ofthe spine. In one embodiment, a sagittal Cobb angle, a coronal Cobbangle, and an axial rotation is displayed on the display of the remotestation. During the spine manipulation the surgeon can store or freeze aspine shape in-situ. The in-situ spine shape will include the metricsgenerated in real-time that characterize the spine shape. Typically, thesurgeon stores the in-situ spine shape if it produces a desired outcome.The in-situ spine shape and metrics can be displayed and compared to thepre-operative plan spine shape and metrics thereby allowing the surgeonto see differences after the subjective analysis of the spine.

In a step 266, the pre-operative defined rod shape is reviewed. Thesurgeon has both quantitative measurement data and subjective datarelated to the spine. The surgeon can then determine if thepre-operative defined rod shape results in a correct spine outcome. Therod shape can be modified to meet the spine outcome based on thereal-time metrics. The new rod shape can be formed and coupled to thepedicle screws. The shape of the spine can then be measured to determineif the new rod shape achieves the outcome desired by the surgeon.

Referring briefly to FIG. 18, an example of a rod 298 coupled to thespine to change the spine contour as disclosed hereinabove is provided.In the example, a lumbar region of the spine is illustrated. The lumbarregion is in view of the camera of the spine measurement system. Thelumbar region comprises L1 vertebra 280, the L2 vertebra 282, L3vertebra 284, L4 vertebra 286, L5 vertebra 288, and Sacrum 290. Apedicle screw 292 couples to L2 vertebra 282. A pedicle screw 294couples to L3 vertebra 284. A pedicle screw 296 couples to L4 vertebra286. A rod 298 is shaped to achieve the desired outcome for the spinesurgery. As disclosed above, the rod shape can be the pre-operative planrod shape or a modified rod shape based on quantitative measurement datafrom the spine measurement system and the subjective assessment of thesurgeon. Rod 298 is held in place by pedicle screw extenders 300, 302,and 304. Pedicle screw extenders 300, 302, and 304 are respectivelycoupled to pedicle screws 292, 294, and 296. In general, a head of apedicle screw is designed to receive a rod. A pedicle screw extenderscrews into a corresponding head of a pedicle screw and clamps down onthe rod. The spine is forced to take the shape of the rod. The rod canforcibly cause vertebra rotation.

Pedicle screw extenders 300, 302, and 304 are in the field of view ofthe camera. Information related to pedicle screw extenders 300, 302, and304 is stored in the remote station. For example, the dimensions andshape of a pedicle screw extender is known. The remote stationapplication using computer vision can recognize pedicle screw extenders300, 302, and 304. Pedicle screw extenders 300, 302, and 304 can havetwo-dimensional or three-dimensional images on an exterior surface tosupport recognition. Moreover, the remote station application candetermine a relative position of pedicle screw extenders 300, 302, and304 in 3D space. The remote station also has information related topedicle screws 292, 294, and 296. The information includes the dimensiondata of each pedicle screw and data related to each pedicle screwinstallation such as the point of entry on the vertebra and thetrajectory into the vertebra. Thus, the remote station can recognize apedicle screw extender position and interpolate back to the location andtrajectory of a corresponding pedicle screw to determine a position of avertebra to which the pedicle screw extender couples. The remote stationcan do this in real-time for each pedicle screw extender in the field ofview. The information can be used with the initial spine shapemeasurements using fluoroscope images to provide a spine image andquantitative measurement relating to the spine shape with rod 298 inplace.

FIG. 19 is a block diagram 320 of a method of pedicle screw placement inaccordance with an example embodiment. The method can use spinemeasurement system 10 of FIG. 1 to drill a path in a vertebra to place apedicle screw. Spine measurement system 10 supports locating a point ofentry in the vertebra and the angle of drilling at the point of entry.Moreover, the method can be practiced with more or less steps and is notlimited to the order of steps shown. The method is not limited to thespine example but can be used for hip, knee, shoulder, ankle, elbow,spine, hand, wrist, foot, bone, and musculoskeletal system.

In a step 322, the camera is directed at the spine. More specifically,the targeted vertebra is in a field of view of the camera including theregion where a pedicle screw is being placed. The camera can focus onthe targeted vertebra to view the bone features in detail. The targetedvertebra is identified by the remote station application using computervision and can be shown on the display of the remote station. Thesurgeon can verify that the targeted vertebra on the display of theremote station corresponds to the vertebra in which the pedicle screw isbeing placed. In one embodiment, the remote station application forpedicle screw placement uses computer vision to identify features orlandmarks that provide a reference for locating specific regions on thetargeted vertebra.

In a step 324, information is retrieved from the remote stationdetailing the location for pedicle screw placement in relation to thelandmarks on the targeted vertebra. The remote station application usingcomputer vision can identify each vertebra of the spine. The computervision finds features or landmarks specific to each vertebra. Thefeatures or landmarks used by computer vision are selected to be aunique combination for the targeted vertebra yet the features andlandmarks can be used to identify the same vertebra over the largephysical variations that occur in nature. The entry point location forthe targeted vertebra is stored in memory of the remote station. Theinformation describing the entry point location of the targeted vertebrarelates to the identified features and landmarks by the remote stationapplication. Similarly, the trajectory in which to align the drill foreach vertebra at the point of entry is stored in memory of the remotestation and can be related to the identified features and landmarks. Inone embodiment, the point of entry can be displayed on the display ofthe remote station by a circle, crosshair, or other identifier to locatethe spot where drilling should occur.

Referring briefly to FIG. 20, a lateral view of the lower lumbar regionis illustrated. The lumbar region comprises L1 vertebra 280, the L2vertebra 282, L3 vertebra 284, L4 vertebra 286, L5 vertebra 288, andsacrum 290. A pedicle screw 292 couples to L2 vertebra 282. A pediclescrew 296 couples to L4 vertebra 286. L3 vertebra 284 is being drilledfor pedicle screw placement. Spine measurement system 10 of FIG. 1 canbe used to place pedicle screws. A drill 340 is shown approaching L3vertebra 284. Drill 340 has a trajectory 342 from the lateral view thathas been identified as an optimal path for pedicle screw placement forL3 vertebra 284. In general, the remote station application usingcomputer vision monitors the position of drill 340 in real-time. Theremote station application directs drill 340 to trajectory 342 usingvisual, audio, and haptic queues.

Referring briefly to FIG. 21, a transverse view of L3 vertebra 284 isillustrated. Spine measurement system 10 of FIG. 1 can be used forpedicle screw placement. L3 vertebra 284 is being drilled for pediclescrew placement. A point of entry 344 is identified by the remotestation for optimal placement of a pedicle screw. Drill 340 is shownapproaching L3 vertebra 284 and will contact point of entry 344. Drill340 has a trajectory 346 from the transverse view that has beenidentified as an optimal path for pedicle screw placement for L3vertebra 284. FIG. 20 and FIG. 21 illustrates that the trajectory can becomplex. The remote station application using computer vision canmonitor the position of drill 340 in 3D space and direct the trajectoryfor optimal placement as viewed in both the transverse and lateralplanes as disclosed in FIG. 20 and FIG. 21.

In a step 326 the tool is guided to a point of entry of the targetedvertebra. In one embodiment, the remote station application usescomputer vision to locate and identify the position of the tool inrelation to the targeted vertebra. The tool can also include positionsensing technology such as accelerometers, gyroscopes, globalpositioning system, infra-red, optical, or acoustic to support placementof the tool. In the example, an opening will be drilled to locate apedicle screw in the targeted vertebra. The drill tip can be identifiedby the remote station application using computer vision and directed tothe point of entry on the targeted vertebra. The display of the remotestation can also be used to support locating the tool to the point ofentry. Visual aids can be used in the targeting process to locate thetool to the point of entry. Similarly, the tool is guided to have acorrect trajectory. The tool needs to be placed in the properorientation at the point of entry on the targeted vertebra beforedrilling a path. As mentioned previously, the remote station retrievesthe stored information related to the tool trajectory for placing apedicle screw in the targeted vertebra. In one embodiment, the remotestation application using computer vision recognizes the position andtrajectory of the tool in relation to the identified landmarks andfeatures of the targeted vertebra. The sensing technology in the toolcan also support aligning the tool in the correct trajectory. Visualaids on the display of the remote station can be used in aligning thetool to the correct trajectory. Once the tool is placed at point ofentry at the correct trajectory an opening is formed in the targetedvertebra corresponding to where the pedicle screw is placed. In theexample, the targeted vertebra is drilled to a predetermined depth.Alternatively, a wire could be placed through the bone to apredetermined depth.

In a step 328 the tool is monitored. The tool is in the field of view ofthe camera. The remote station application using computer visionrecognizes and can track the position of the tool in 3D space as itrelates to the spine. The remote station application tracks the tool inreal-time within the field of view of the camera. The remote station canindicate the point of entry on the targeted vertebra. The remote stationapplication can further identify that the tool couples to the point ofentry on the targeted vertebra.

In a step 330, information related to a correct trajectory of the toolis stored on the remote station. The information is retrieved and usedby the remote station application in comparing an alignment of the toolto the correct trajectory in support of achieving a correct toolalignment at the point of entry on the target vertebra. The trajectoryof the tool can be indicated in real-time. The trajectory can beindicated on the tool or the display of the remote station. The remotestation using computer vision recognizes the tool and the trajectory ofthe tool. Moreover, the remote station supports aligning the tool to thecorrect trajectory.

In a step 332, feedback can be provided to support aligning the tool atthe point of entry of the targeted vertebra. Similarly, feedback can beprovided to support placing the tool at the correct trajectory. Thefeedback can be visual, audible, or haptic. For example a simple greenlight or red light can be used in locating the tool to the point ofentry on the targeted vertebra. Information related to the direction tomove the tool can be provided on the display of the remote station untilthe green light is received.

FIG. 22 is an illustration of spine measurement system 10 configured tosupport pedicle screw placement in accordance with an exampleembodiment. In general, each vertebra of the spine is different.Similarly, an optimal pedicle screw placement is different for eachvertebra of the spine. Spine measurement system 10 supports placing atool at a point of entry of a targeted vertebra and placing the tool atthe correct trajectory to drill a path for pedicle screw placement.Optical measurement probe 12 includes a camera that is configured tohave a spine in a field of view. Optical measurement probe 12 can bemounted to a stable object that allows a minimally impeded view of thespine and more specifically a targeted vertebra for pedicle screwplacement. The position of optical measurement probe 12 can bereferenced prior to starting a procedure. The camera of opticalmeasurement probe 12 can send image data related to static objects toremote station 20. The remote station application using computer visioncan locate and identify a position of three or more static objects. Forexample, on a wall in the field of view of the camera there are staticobjects such as a cabinet 360, a light switch, 362, an outlet 364, or adoor knob 366. The static objects in the field of view of the cameracannot move during the surgery. An origin of the coordinate system isconfigured to be referenced to at least three static objects in thefield of view of the camera. In one embodiment, the origin of thecoordinate system of spine measurement system 10 for pedicle screwinstallation is configured to be located in the camera of opticalmeasurement probe 12. The position of the static objects will havedifferent position when referenced to the origin if the camera is movedfrom its position and orientation. A visual, audible, or haptic warningis provided when movement of the camera is detected. The warning isprovided to indicate that movement has occurred but also to determine ifit was the camera or something else had moved. The spine measurementsystem 10 can also turn off any tool that can be affected by themovement. For example, the tool could be positioned incorrectly bysystem 10 when movement is detected. Turning the tool off prevents anirreversible error from occurring until the problem is resolved. In oneembodiment, the remote station application can compensate by adding anoffset that accounts for a camera movement so all measurements remainedlinked to the original coordinate system. Conversely, movement can occurthat requires the coordinate system to be reset and the process startedover to locate the tool.

Remote station 20 is configured to receive image data from the camera.In one embodiment, the camera in optical measurement probe 12 is coupledby cable to remote station 20. Alternatively, the camera is coupledwirelessly to remote station 20. The spine is exposed such that thelandmarks and features of the vertebra can be viewed by the camera.Remote station 20 is configured to use computer vision to catalogvertebrae in the field of view of the camera. Remote station 20 can beconfigured to receive confirmation from a user that the cataloguedvertebrae in the field of view are correctly identified. The remotestation application uses information stored in the remote stationrelated to features or landmarks of each vertebra to recognize from theimage data what vertebra are in the field of view. In one embodiment,the remote station includes a lookup table configured to providereference landmarks or features.

The remote station identifies the targeted vertebra for pedicle screwinstallation and can identify it on display 22 of remote station 20. Thesurgeon can verify that the vertebra identified on display 22 is thetargeted vertebra for pedicle screw installation. In one embodiment, thetool can be configured to penetrate bone of the targeted vertebra. Forexample, a drill is commonly used to form a path in the bone that can betapped to receive the pedicle screw. Remote station 20 using computervision is configured to locate a point of entry of the targetedvertebra. Remote station 20 has stored information relating to a pointof entry for pedicle screw placement for each vertebra. The storedinformation can be on a lookup table having locations of points of entryfor each vertebra. The location of the point of entry can be related tothe features and landmarks used to identify a vertebra. The remotestation application identifies features and landmarks on the targetedvertebra and locates the point of entry using the information stored inthe remote station 20. Furthermore, remote station 20 has storedinformation related to a trajectory to penetrate the targeted vertebra.In one embodiment, the remote station includes a lookup table related tothe trajectory of the tool for the targeted vertebra. Remote station 20is configured to direct the trajectory of the tool in the targetedvertebra. The camera sends image data of the surgeon directing the toolto the point of entry in real-time. The remote station application usingcomputer vision locates a position of the tool in real-time andidentifies when the tool has a correct trajectory for the targetedvertebra and is coupled to the point of entry.

In one embodiment, the tool is a drill. The remote station applicationcan recognize a position of the drill and drill tip using computervision. The drill can include electronic circuitry to communicate withthe remote station. Furthermore, the drill can include position sensorsto provide trajectory or location information. The position sensors inthe drill can be used to support the remote station application indirecting the drill to the point of entry and the correct trajectory forthe targeted vertebra. The tool can be configured to provide feedbackrelated to aligning the tool to the point of entry of the targetedvertebra and to the trajectory of the tool at the point of entry. Thefeedback to the user can be visual, audible, or haptic. The feedbacksimplifies positioning the tool at the point of entry and at the correcttrajectory. Remote station 20 can also provide feedback to the surgeon.The camera and remote station are configured to monitor the toolposition and trajectory in real-time. The display 22 of remote station20 can also provide visual, audible, or haptic feedback to the surgeonto support positioning the tool to the point of entry and positioningthe tool for correct trajectory. The feedback can provide feedbackrelated to correct and incorrect placement.

FIG. 23 is an illustration of remote station 20 tracking a position ofthe tool in accordance with an example embodiment. The tool position andtrajectory is displayed on display 22 in real-time. The remote stationapplication uses computer vision to identify and track the position ofthe tool in 3D space using image data received from the cameramonitoring the spine. In the example, a posterior view of the lumbarregion is displayed on display 22. The lumbar region comprises L1vertebra 280, the L2 vertebra 282, L3 vertebra 284, L4 vertebra 286, L5vertebra 288, and sacrum 290. In one embodiment, the tool is a drill.The drill is used to drill a path in the targeted vertebra to install apedicle screw. The point of entry is a location in the targeted vertebrato install a pedicle screw. In one embodiment, display 22 includes acircle 360 that encompasses at least a portion of the targeted vertebra.A cross-hair 362 in circle 360 identifies the location of point of entryfor the drill. The point of entry is located by the remote stationapplication identifying features and landmarks of the targeted vertebraand retrieving stored information relating the point of entry to theidentified features or landmarks in the field of view of the camera.

The drill can be represented by a cylinder on display 22 of the remotestation 20. The center of the drill can be represented by a cross-hairon the cylinder. As mentioned, the remote system application receivesimage data and tracks a position of the drill as it is moved by thesurgeon. Multiple cylinders are shown to indicate movement of the drillto the point of entry. Movement of the drill is indicated by cylinder364, cylinder 366, and cylinder 368. Note that the indicated movement ofcylinders 364, 366, and 368 has the drill moving towards the point ofentry. The drill not only has to align to the point of entry but thedrill also has a trajectory alignment. In one embodiment, the trajectoryalignment can be represented by a circle formed around cross-hair 362.The circle is approximately the same diameter or larger than cylinders364, 366, and 368.

Cylinder 364 represents the drill being the furthest from the point ofentry. The drill trajectory is also misaligned from the correcttrajectory. The trajectory is indicated by the three dimensionalrepresentation of cylinder 364. Note that a cylinder wall of cylinder364 is visible on display 22. In one embodiment, an image of a cylinderon display 22 appearing as a circle without a cylinder wall visible willhave the correct trajectory. The drill is moving closer to the point ofentry, which is represented by cylinder 366 in display 22. Cylinder 366is closer to the point and has less cylinder wall exposed than cylinder364. The reduction in the exposed cylinder wall indicates that thetrajectory of the drill is moving towards the correct trajectory.Cylinder 368 represents the drill almost at the point of entry. Thecross-hair at the point of entry almost aligns with the cross-hair oncylinder 368. Similarly, the cylinder wall of cylinder 368 almost cannotbe seen. In other words, cylinder 368 is almost at the correcttrajectory which is represented by cylinder 368 viewed as a circle ondisplay 22 of remote station 20. The use of visual, audible, or hapticfeedback such as beeping or vibrating could be used to further enhancealigning the tool. For example, a rapid audible beeping can indicatethat the device is not close to the point of entry. The audible beepingcould slow down or quit when the point of entry is aligned to thedevice. Similarly, the drill could be made to vibrate when thetrajectory alignment is incorrect. The vibration could be reduced as thedrill nears the correct trajectory. A green light indicator could flashwhen the correct trajectory is reached and the vibration stops. Theseare just examples as other types of feedback can be provided thatsupports alignment to the point of entry and alignment to the correcttrajectory.

FIG. 24 depicts an exemplary diagrammatic representation of a machine inthe form of a spine measurement system 450 within which a set ofinstructions, when executed, may cause the machine to perform any one ormore of the methodologies discussed above. In some embodiments, themachine operates as a standalone device. In some embodiments, themachine may be connected (e.g., using a network) to other machines. In anetworked deployment, the machine may operate in the capacity of aserver or a client user machine in server-client user networkenvironment, or as a peer machine in a peer-to-peer (or distributed)network environment.

The machine may comprise a server computer, a client user computer, apersonal computer (PC), a tablet PC, a laptop computer, a desktopcomputer, a control system, logic circuitry, a sensor system, an ASIC,an integrated circuit, a network router, switch or bridge, or anymachine capable of executing a set of instructions (sequential orotherwise) that specify actions to be taken by that machine. It will beunderstood that a device of the present disclosure includes broadly anyelectronic device that provides voice, video or data communication.Further, while a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines that individually orjointly execute a set (or multiple sets) of instructions to perform anyone or more of the methodologies discussed herein.

System 450 may include a processor 452 (e.g., a central processing unit(CPU), a graphics processing unit (GPU, or both), a main memory 454 anda static memory 456, which communicate with each other via a bus 458.System 450 may further include a video display unit 460 (e.g., a liquidcrystal display (LCD), a flat panel, a solid state display, or a cathoderay tube (CRT)). System 450 may include an input device 462 (e.g., akeyboard), a cursor control device 464 (e.g., a mouse), a disk driveunit 466, a signal generation device 468 (e.g., a speaker or remotecontrol) and a network interface device 472.

The disk drive unit 466 can be other types of memory such as flashmemory and may include a machine-readable medium 474 on which is storedone or more sets of instructions (e.g., software 470) embodying any oneor more of the methodologies or functions described herein, includingthose methods illustrated above. Instructions 470 may also reside,completely or at least partially, within the main memory 454, the staticmemory 456, and/or within the processor 452 during execution thereof bythe system 450. Main memory 454 and the processor 452 also mayconstitute machine-readable media.

Dedicated hardware implementations including, but not limited to,application specific integrated circuits, programmable logic arrays andother hardware devices can likewise be constructed to implement themethods described herein. Applications that may include the apparatusand systems of various embodiments broadly include a variety ofelectronic and computer systems. Some embodiments implement functions intwo or more specific interconnected hardware modules or devices withrelated control and data signals communicated between and through themodules, or as portions of an application-specific integrated circuit.Thus, the example system is applicable to software, firmware, andhardware implementations.

In accordance with various embodiments of the present disclosure, themethods described herein are intended for operation as software programsrunning on a computer processor. Furthermore, software implementationscan include, but not limited to, distributed processing orcomponent/object distributed processing, parallel processing, or virtualmachine processing can also be constructed to implement the methodsdescribed herein.

The present disclosure contemplates a machine readable medium containinginstructions 470, or that which receives and executes instructions 470from a propagated signal so that a device connected to a networkenvironment 476 can send or receive voice, video or data, and tocommunicate over the network 476 using the instructions 470. Theinstructions 470 may further be transmitted or received over a network476 via the network interface device 472.

While the machine-readable medium 466 is shown in an example embodimentto be a single medium, the term “machine-readable medium” should betaken to include a single medium or multiple media (e.g., a centralizedor distributed database, and/or associated caches and servers) thatstore the one or more sets of instructions. The term “machine-readablemedium” shall also be taken to include any medium that is capable ofstoring, encoding or carrying a set of instructions for execution by themachine and that cause the machine to perform any one or more of themethodologies of the present disclosure.

The term “machine-readable medium” shall accordingly be taken toinclude, but not be limited to: solid-state memories such as a memorycard or other package that houses one or more read-only (non-volatile)memories, random access memories, or other re-writable (volatile)memories; magneto-optical or optical medium such as a disk or tape; andcarrier wave signals such as a signal embodying computer instructions ina transmission medium; and/or a digital file attachment to e-mail orother self-contained information archive or set of archives isconsidered a distribution medium equivalent to a tangible storagemedium. Accordingly, the disclosure is considered to include any one ormore of a machine-readable medium or a distribution medium, as listedherein and including art-recognized equivalents and successor media, inwhich the software implementations herein are stored.

Although the present specification describes components and functionsimplemented in the embodiments with reference to particular standardsand protocols, the disclosure is not limited to such standards andprotocols. Each of the standards for Internet and other packet switchednetwork transmission (e.g., TCP/IP, UDP/IP, HTML, HTTP) representexamples of the state of the art. Such standards are periodicallysuperseded by faster or more efficient equivalents having essentiallythe same functions. Accordingly, replacement standards and protocolshaving the same functions are considered equivalents.

FIG. 25 is an illustration of a communication network 400 formeasurement and reporting in accordance with an exemplary embodiment.Briefly, the communication network 400 expands broad data connectivityto other devices or services. As illustrated, the measurement andreporting system 422 can be communicatively coupled to thecommunications network 402 and any associated systems or services.

As one example, measurement system 422 can share its parameters ofinterest (e.g., angles, load, balance, distance, alignment,displacement, movement, rotation, and acceleration) with remote servicesor providers, for instance, to analyze or report on surgical status oroutcome. This data can be shared for example with a service provider tomonitor progress or with plan administrators for surgical monitoringpurposes or efficacy studies. The communication network 400 can furtherbe tied to an Electronic Medical Records (EMR) system to implementhealth information technology practices. In other embodiments, thecommunication network 400 can be communicatively coupled to HIS HospitalInformation System, HIT Hospital Information Technology and HIM HospitalInformation Management, EHR Electronic Health Record, CPOE ComputerizedPhysician Order Entry, and CDSS Computerized Decision Support Systems.This provides the ability of different information technology systemsand software applications to communicate, to exchange data accurately,effectively, and consistently, and to use the exchanged data.

The communications network 400 can provide wired or wirelessconnectivity over a Local Area Network (LAN) 402, a Wireless Local AreaNetwork (WLAN) 410, a Cellular Network 414, and/or other radio frequency(RF) system. The LAN 402 and WLAN 410 can be communicatively coupled tothe Internet 416, for example, through a central office. The centraloffice can house common network switching equipment for distributingtelecommunication services. Telecommunication services can includetraditional POTS (Plain Old Telephone Service) and broadband servicessuch as cable, HDTV, DSL, VoIP (Voice over Internet Protocol), IPTV(Internet Protocol Television), Internet services, and so on.

The communication network 400 can utilize common computing andcommunications technologies to support circuit-switched and/orpacket-switched communications. Each of the standards for Internet 416and other packet switched network transmission (e.g., TCP/IP, UDP/IP,HTML, HTTP, RTP, MMS, SMS) represent examples of the state of the art.Such standards are periodically superseded by faster or more efficientequivalents having essentially the same functions. Accordingly,replacement standards and protocols having the same functions areconsidered equivalent.

The cellular network 414 can support voice and data services over anumber of access technologies such as GSM-GPRS, EDGE, CDMA, UMTS, WiMAX,2G, 3G, WAP, software defined radio (SDR), and other known technologies.The cellular network 414 can be coupled to base receiver 412 under afrequency-reuse plan for communicating with mobile devices 404.

The base receiver 412, in turn, can connect the mobile device 404 to theInternet 416 over a packet switched link. The internet 416 can supportapplication services and service layers for distributing data from spinemeasurement system 422 to the mobile device 404. Mobile device 404 canalso connect to other communication devices through the Internet 416using a wireless communication channel.

The mobile device 404 can also connect to the Internet 416 over the WLAN410. Wireless Local Access Networks (WLANs) provide wireless accesswithin a local geographical area. WLANs are typically composed of acluster of Access Points (APs) 408 also known as base stations. Thespine measurement system 422 can communicate with other WLAN stationssuch as laptop 406 within the base station area. In typical WLANimplementations, the physical layer uses a variety of technologies suchas 802.11b or 802.11g WLAN technologies. The physical layer may useinfrared, frequency hopping spread spectrum in the 2.4 GHz Band, directsequence spread spectrum in the 2.4 GHz Band, or other accesstechnologies, for example, in the 5.8 GHz ISM band or higher ISM bands(e.g., 24 GHz, etcetera).

By way of the communication network 400, the spine measurement system422 can establish connections with a remote server 418 on the networkand with other mobile devices for exchanging data. The remote server 418can have access to a database 420 that is stored locally or remotely andwhich can contain application specific data. The remote server 418 canalso host application services directly, or over the internet 416.

FIG. 26 is an illustration of spine measurement system 10 in accordancewith an example embodiment. In general, a surgeon bends a rod to alter ashape of the spine. The surgeon typically has a pre-operative plan andtarget metrics to determine if the rod shape achieves a desired spineoutcome. The bending of rod 502 is subjective, and is based on theobservations of the surgeon prior to and during surgery. Rod 502 canhave a complex shape. Prior to bending rod 502, pedicle screws areanchored to vertebra of the spine. Rod 502 is coupled to the pediclescrews in the spine to forcibly change a position of the vertebrathereby changing the spine contour.

Spine measurement system 10 can be used to measure a contour, profile,or shape of rod 502 that is non-linear in shape. An obvious benefit ofmeasuring rod 502 is that metrics related to the spine outcome can becalculated from the measurement of rod 502. Alternatively, measurementsof rod 502 can be used in conjunction with a spine model generated fromquantitative measurements in spine measurement 10 to show how the spineshape is altered on display 22 and similarly calculate metrics relatedto the spine shape. For example, the metrics generated from themeasurement of rod 502 can be Cobb angles and axial rotation asdetermined by the bends in rod 502 and the locations of the knownlocations of the pedicle screws in the vertebra. The metrics can be usedto determine if rod 502 needs to be modified to achieve the spineoutcome. Alternatively, remote station 20 can simulate how changing rod502 position affects calculated metrics. Remote station 20 can be usedto determine placement of rod 502 to best achieve the desired outcome orto suggest modifications to rod 502 if the outcome is not achieved.

The rod measurement system comprises encoded collar 504, a camera, and aremote station 20. Encoded collar 504 includes markings that relate aregion of collar 504 to a region of rod 502. In one embodiment, encodedcollar 504 corresponds to an angular orientation of rod 502. In theexample, encoded collar 504 is fitted on rod 502 such rotating rod 502also rotates collar 504. Encoded collar 504 can be an interference fitor can attach to the rod in a manner where encoded collar 504 does notshift or change position in relation to the rod. In the example, encodedcollar is a degree wheel having markings that indicate a position of thewheel. A full rotation of rod 502 corresponds to 360 degrees on encodedcollar 504. A portion of rod 502 from a 2D image taken by the camera isrelated to the degree markings indicated on encoded collar 504. Thus,each 2D image taken by the camera provides a contour of rod 502 and arod position that is indicated on encoded collar 504 by the identifyingmarkings (e.g. degrees of the degree wheel). The degree wheel is anexample of markings that can be used to identify a rod position. Ingeneral, the identifying markings can be of any form that can beidentified from an image. For example, bar code can be used to identifyportions of encoded collar 504. Bar code can be read from a 2D image inan automated process and used to identify the position or angularorientation of rod 502. In the example, the camera is in opticalmeasurement probe 12 as part of a spine measurement system. The cameracouples to remote station 20 and is configured to provide images ofencoded collar 504 and rod 502. In one embodiment, remote station 20 isa processor based system that is configured to run software. Remotestation 20 includes image processing software and is configured toprocess the images and generate quantitative measurement data related tothe rod shape in 3D. Remote station 20 is configured to recognize rod502, encoded collar 504, and the plurality of markings on encoded collar504. In the example, the dimensions of encoded collar 504 are providedto the remote station. The known dimensions of encoded collar 504 areused as a scale reference for the remote station to relate one image toanother such that rod 502 and encoded collar 504 are substantial equalin each image. The known dimensions of encoded collar 504 are also usedas a reference to measure dimensions of other objects in the image suchas rod 502. In particular, remote station 20 is configured to receiveimages taken while rod 502 is rotated at least 360 degrees. The camerain optical measurement probe 12 is configured to take a plurality ofimages while rod 502 is rotated at least 180 degrees. Remote station 20receives the images and processes the images to generate quantitativemeasurement data related to a shape of rod 502 in three-dimensions (e.g.3D). The measurement comprises 2D portions of rod 502 approximating 180degree view of rod 502 that are summed together to generate a 3Dmeasurement of the rod contour or rod shape. In one embodiment, rod 502is rotated 360 degrees or more.

In the example, a distal end of rod 502 couples to a platen 506. Asshown, encoded collar 504 is coupled to rod 502. Alternatively, platen506 can have markings similar to encoded collar 504 thereby eliminatingthe need for encoded collar 504. Platen 506 includes a spindle and abearing that supports rotation. In one embodiment, platen 506 canrotated by hand. Optical measurement probe 12 will capture one or morefull rotations of rod 502 and collar 504. In another embodiment, platen506 can be rotated by a motor coupled to platen 506. Using a motorizedplaten 506, rod 502 can be rotated at a constant speed and not subjectto variations that could occur when spinning by hand.

The camera in optical measurement probe 12 takes a number of imageswhile rod 502 is spinning. In general, the camera in optical measurementprobe 12 can provide high resolution images that supports reading themarkings on collar 504 on each image. As an example, the rod 502 can bespun at one revolution per second. Optical measurement probe 12 can takegreater than 30 images per second allowing each image to correspond to12 degrees of rod 502. The images are provided to remote station 20. Ingeneral, the number of images and the speed at which rod 502 is spun canvary but needs to be sufficient to generate the three dimensionalprofile of rod 502 from a plurality of images. Remote station 20 canprovide measurements of rod 502 or related metrics from the measurementof rod 502. In one embodiment, metrics related to how rod 502 willaffect the curvature of the spine are calculated by remote station 20and displayed on display 22. For example, sagittal Cobb angle, coronalCobb angle, and axial rotation can be provided corresponding to measuredrod 502 to allow the surgeon to determine if rod 502 meets the goals ofthe surgery.

FIG. 27 is an illustration of spine measurement system 10 in accordancewith an example embodiment. Optical measurement probe 12 includes acamera configured to provide images to remote station 20. Encoded collar504 couples to a distal end of rod 502. Encoded collar 504 includes aknob 508 that is a finger hold. Rod 502 and encoded collar 504 are in afield of view of the camera of optical measurement probe 12. Knob 508can be held in the thumb and forefinger and rotated thereby rotating rod502. Optical measurement probe 12 is configured to capture apredetermined number of images over a single complete rotation of rod502. The identifying markings on encoded collar 504 facing the camera inoptical measurement probe 12 indicate the angular orientation of rod502. In general, the number of images and the speed at which rod 502 canbe spun will vary depending on the user but needs to be sufficient togenerate the three dimensional profile of rod 502 from a plurality ofimages. Typically, more than ten 2D images per 360 degrees of rotationare needed to generate a 3D measurement of rod 502. In the example,thirty 2D images per 360 degrees of rotation are summed to generate a 3Dmeasurement of rod 502. The images are provided to remote station 20.Remote station 20 can provide measurements of rod 502 or relatedmetrics. As shown on display 22, a sagittal Cobb angle, coronal Cobbangle, and axial rotation is provided from the measurement of rod 502 onremote station 20. The surgeon can determine if the providedmeasurements corresponding to the shape of rod 502 meet the goals andobjectives of the surgery. The surgeon can modify rod 502 to change thespine shape to meet the desired outcome and rod 502 can be re-measuredto provide metrics indicating the change a change in outcome.

FIG. 28 is side view image 510 and top view image 512 of rod 514 andencoded collar 516 in accordance with an example embodiment. In theexample, a distal end of rod 514 is coupled within an opening in encodedcollar 516. A finger hold 518 extends from a main body of encoded collar516. Similar, to the manual method of rotation described hereinabove,finger hold 518 can be held by thumb and fore finger (or othervariation) to rotate rod 514 and encoded collar 516. In one embodiment,the camera is held or mounted so the field of view captures the entirelength of rod 514 and encoded collar 516. For example, opticalmeasurement probe 12 is held such that the lens of the camera isperpendicular to a surface 564 of encoded collar 516. The camera shouldbe able to capture the identifying markings on the side, top surface516, or both of encoded collar 516.

The dimensions of encoded collar 516 are known. For example, the heightand diameter of the main body of encoded collar 516 is known andprovided to remote station 20. Similarly, the height and diameter offinger hold 518 is known. Dimensions of rod 514 can calculated bycomparing and scaling to the known dimensions of encoded collar 516 onan image. In one embodiment, encoded collar 516 has identifying markingson around a periphery of the main body of encoded collar 516. In theexample, bar codes are used to identify sections of encoded collar 516and a corresponding portion of rod 514. The periphery of the main bodyshould be visible to the camera as it is spun. A top surface of encodedcollar 516 can also included identifying markings that correspond toportions of rod 514. As mentioned the markings on encoded collar 516 areunique in each section and can be recognized by the remote station fromimages it receives.

Image 510 and image 512 relate to a single image provided by the cameraof optical measurement probe 12. It should be noted that approximately360 degrees of 2D images of rod 514 are provided by the camera ofoptical measurement probe 12. Each of the 2D images is identified by aunique mark or identifier found on a side or a surface of encoded collar516 that represents an arc of rod 514. Each image received by the remotestation is processed similar to image 510 and 512. The surgeon may notbe able to keep rod 514 and encoded collar 516 a constant distance fromthe camera in optical measurement probe 12. In general, the remotestation can detect objects in an image. The remote station also has theability to scale and rotate image. In one embodiment, the remote stationscales the images to each other such that the encoded collar 516 are thesame size in each image. The remote station can also rotate each imageso they are viewed from the same point of reference. The shape ofencoded collar 516 can support the translation process. Encoded collar516 can have sloped edges and other distinguishing features that can becompared and assessed from image to image by the remote station. Thedimensions of the distinguishing features will also be known which willsupport manipulation of each image to make the encoded collar 516 thesame in each image and whereby the dimensions of rod 514 aresubstantially equivalent and the profile can measured. The images canthen be summed together to produce a 3D measurement of rod.

The remote station will project a measurement grid on image 510 tomeasure a contour of rod 514. The measurement grid comprises a centralvertical axis 562 and a plurality of horizontal spaced lines that areperpendicular to central vertical axis 562. The horizontal spaced linescomprise lines 520, 522, 524, 526, 528, 530, 532, 534, 536, 538, and540. Central vertical axis 562 is perpendicular to a top surface 564 ofencoded collar 516. The plurality of spaced lines intersect rod 514 atdifferent points. The points of intersection are points 542, 544, 546,548, 550, 552, 554, 556, 558, and 560. The number of number ofhorizontal lines and the intersections points can be more or less thanshown. As mentioned previously, the dimensions of encoded collar areknown and provided to the remote station. The dimensions of rod 514 canbe determined by referencing the known dimensions of encoded collar 516to the image rod 514. For example, horizontal lines 520, 522, 524, 526,528, 530, 532, 534, 536, 538, and 540 the spacing between the horizontallines can represent a distance of 1 centimeter. The horizontal linespacing representing 1 centimeter on image 510 can be scaled from knowndimensions of encoded collar 516 on image 510 and using those dimensionsto scale the line spacing. Thus, as shown, the height of rod 514 in theexample is approximately 10 centimeters. Similarly, the distance ofpoints 542, 544, 546, 548, 550, 552, 554, 556, 558, and 560 from centralvertical axis 562 can be scaled and measured by the remote station fromthe known dimensions of encoded collar 516. Curve fitting can be usedbetween points to generate a shape that approximates rod 514 if needed.As mentioned, encoded collar 516 is rotated which also rotates rod 514.Points 542, 544, 546, 548, 550, 552, 554, 556, 558, and 560 are measuredby the remote station throughout a 360 degree rotation of rod 514 fromthe images provided. The remote station will also log or note keyfeatures from the images. For example, the remote station can log andidentify where a maximum occurs for each point. The location or positionof the point maximum is identified by an angle or the identified arcfrom encoded collar 516. In general, a table is generated indicating adisplacement from central vertical axis 562 of points 542, 544, 546,548, 550, 552, 554, 556, 558, and 560 on each 2D image and thecorresponding position or angular orientation of rod 514 from theidentified marking on encoded collar 516 as measured by the remotestation. The identified marking on the encoded collar 516 corresponds toan angle. Thus, we can translate to polar notation since we have theradius of the points from the central vertical axis 562 and the anglerepresented by the marking on encoded collar 516 for the 2D image.

In one embodiment, the table is stored in polar notation that provides aradius from central vertical axis 562 at an angle for each point todefine a profile of rod 514. Image 512 represents a top view lookingdown on rod 514 and encoded collar 516. The circles represent angularchange and the orthogonal lines represents the radius from the centralvertical axis 562. The table generated by the remote station can betranslated to different coordinate systems if required. In oneembodiment, the polar coordinates in the table can be translated toCartesian coordinates. A 3D model of rod 514 can be constructed from thedata points in the table. The remote station can use a spline algorithmto interpolate the shape of rod 514 from the table of measured datapoints. The spline is fitted to the point constellation provided fromthe table of data points generated from the 2D images. Metrics can begenerated from the shape of rod 514 in conjunction with other datastored in the remote station. In one embodiment, the remote station willhave the positions of each pedicle screw in the spine and positions ofthe endplates of each vertebra relative to each pedicle screw. A Cobbangle can be calculated using the information stored in the remotestation. The 3D shape of the rod is known from the spline interpolationof the measured data points from the 2D images. The pedicle screws arethe coupling points between the rod and the vertebra. The locations ofthe pedicle screw are known and can be located on the rod. The positionof the vertebrae is known on the rod from the pedicle screws. Theendplate to pedicle screw relationship is also known by the remotestation. Thus, the Cobb angle can be calculated from the position of themost proximal and distal endplates of the region of interest.

FIG. 29 is a block diagram 600 of a method of measuring a shape of a rodfor a spine in accordance with an example embodiment. A method ofmeasuring a shape of the rod using a spine measurement system isdisclosed. The rod is typically bent by the surgeon. The measured rodshape can be converted to one or metrics that allows a surgeon todetermine if the measured rod shape will meet a desired outcome for thepatient prior to installation. The rod can be adjusted if the spineoutcome is not met by the measured rod shape. The method can bepracticed with more or less steps and is not limited to the order ofsteps shown. The method is not limited to the spine example but can beused for hip, knee, shoulder, ankle, elbow, spine, hand, wrist, foot,bone, and musculoskeletal system. The components listed in the methodcan be referred to and are disclosed in FIG. 5. In a block 602, a shapeof the rod is measured. In one embodiment, the process of measuring therod is automated where the user requires no measurement tools or takesany measurements. Locations of the pedicle screws in a spine are known.The pedicle screw locations can be stored during a pedicle screwinstallation process that can be performed using the spine measurementsystem as disclosed hereinabove. The locations of the pedicle screw areapplied to the shape of the rod. The vertebra are positioned to the rodby locating the predetermined points where the pedicle screws couple tothe rod. The shape of the spine is determined by the shape of the rodand where the pedicle screws couple to the rod. In one embodiment, thelocation of the pedicle screw and end plates of the vertebra are knownand stored in a remote station. The remote station is also configured tomeasure a shape of the rod. A Cobb angle can be generated thatcorresponds to the shape of the rod. In one embodiment, the remotestation uses quantitative measurements corresponding to the shape of therod and pedicle screw locations to calculate the Cobb angle. The surgeonwill use the Cobb angle or other metrics to determine if the rod shapewill meet the goals of the surgery.

In a block 604, an encoded collar is coupled to the rod. The encodedcollar includes markings that correspond to a position of the rod. Therod is rotated at least 360 degrees. The rod is rotated in a field ofview of a camera. The camera captures a plurality of images of the rodand encoded collar. In general, the camera can capture a plurality ofimages as the rod rotates 360 degrees. The 2D images are used to measurea 3D shape of the rod. A marking is identified on the encoded collar ineach image relating a position or the rod to a rod profile at theposition. The plurality of images of the encoded collar and the rod areprovided to a remote station for processing each image.

In a block 606, the plurality of images are received by the remotestation. In one embodiment, the remote station is configured torecognize the rod, the encoded collar, and a plurality of markings onthe encoded collar. As mentioned previously, a marking corresponds to aposition of the rod in each image. Each image is scaled by the remotestation. In one embodiment, the remote station has the dimensions of theencoded collar. The remote station is configured to scale images. Theremote station scales each image such that the encoded collar issubstantially equal in each image. An orientation of each image istranslated and rotated to have a substantially equal orientation. Theremote station is configured to translate and rotate an orientation ofan image.

In a block 608, a measurement grid is projected on each image. In oneembodiment, the remote station projects the measurement grid on eachimage. A plurality of points are measured on the rod. In one embodiment,the measurement grid intersects the rod at multiple points on eachimage. The intersection of the measurement grid and the rod are theplurality of points on each image. The remote station measures theplurality of points in relation to the measurement grid. For exampleeach point of the plurality of points can be measured from a centralaxis. A marking is identified on the encoded collar for each image. Theposition of the rod for each image is identified by the marking on theencoded collar. The measured positions of the plurality of pointscorresponds to the position of the encoded collar. The position of therod and the measurement of the plurality of points is placed in a table.The remote station is configured to create the table and store themeasurement of the plurality of points and the position of the rod inthe table for each image.

In a block 610, a 3D shape of the rod is interpolated. The remotestation is configured to use a spline algorithm to interpolate the 3Dshape of the rod. A Cobb angle is interpolated from the shape of therod. The remote station is configured to interpolate the Cobb angle fromthe 3D shape of the rod and locations of pedicle screws in the spine. Inone embodiment, the remote station will have locations of the endplatesof each vertebra in relation to the pedicle screws. Thus, the Cobb anglecan be calculated as disclosed hereinabove by interpolating the plane ofa proximal vertebra endplate and the distal vertebra endplate of theregion of interest and measuring the Cobb angle between intersectingplanes.

FIG. 30 is block diagram 700 illustrating using computer vision toidentify or recognize an object in accordance with an exampleembodiment. In general, computer vision can be to recognize an object.In one embodiment, a computer receiving an image can recognize one ormore objects using computer vision software. Similarly, the computervision software can be run in real-time to identify, recognize, andtrack objects relative to each other using a camera providing a livefeed. There are many different computer vision algorithms that can beused to recognize objects or be optimized for a medical application. Forexample, some of the computer vision appearance-based algorithms areedge matching, divide and conquer strategy, grayscale matching, gradientmatching, histograms of receptive field responses, and large modelbases. Similarly, feature based computer vision algorithms can compriseinterpretation trees, hypothesize and text, pose consistency, poseclustering, invariance, geometric hashing, scale-invariant featuretransform (SIFT), and speeded up robust features (SURF). Theappearance-based methods use templates or exemplars of the objects toperform recognition. The feature-based method finds matches betweenobject features and image features. The algorithm used can comprise oneor more aspects of the different algorithms listed above.

A method of recognizing an object using the spine measurement system 10of FIG. 1, FIG. 5, or FIG. 22 is disclosed. The method can be practicedwith more or less steps and is not limited to the order of steps shown.The method is not limited to the spine example but can be used for hip,knee, shoulder, ankle, elbow, spine, hand, wrist, foot, bone, andmusculoskeletal system. The components listed in the method can bereferred to herein below and are disclosed in FIG. 1, FIG. 5 or FIG. 22.In general, a fluoroscope 18 is one source for providing an image of thespine for object recognition such as vertebra endplates or pediclescrews for measurement of angles related to spine shape. Opticalmeasurement probe 12 is another source for providing an image or images.Images are provided to remote station 20 where computer vision softwarecan be used to recognize objects of interest related to the spine. Inone embodiment, remote station 20 is a computer that can be local to theoperating room or be located remotely. Furthermore, remote station 20can recognize, analyze, and generate quantitative measurements inreal-time related to spine shape for use by the surgeon or operatingteam in the operating room.

In a step 702 a region of interest (ROI) is created around a candidateblob. In one embodiment, one or more digital images are provided to thecomputer (e.g. remote station 20) executing computer vision software. Ablob is a portion of a digital image where differences in properties canbe used for detection and where other properties can be similar.Examples of differing properties are brightness or color in an image. Ingeneral, blob detection can be used to detect the region of interestwhere specific objects are being recognized. In the example, theidentified blobs have a high probability of being the objects beingidentified on the digital image. The computer will have information anddata related to the objects to be recognized. Once the region ofinterest is identified further processing can performed. In oneembodiment, object recognition is performed. The object can then betracked after being recognized by the computer.

An example, of recognizing one or more pedicle screws will be used toillustrate a computer vision process for spine measurement system 10.Referring briefly to FIG. 31, a fluoroscope image 720 of a lateral viewof a spine is shown. A digital image of fluoroscope image 720 isprovided to the computer. Pedicle screws 722, 724, and 726 can be seenwithin image 720. In one embodiment, end plates of each vertebra willalso be recognized from fluoroscope image 720. Referring briefly to FIG.32, a binary image 730 of the fluoroscope image is created by thecomputer. The regions of interest are shown in white on binary image730. For example, a threshold is set where pixels having a brightnessabove a threshold are converted to white on binary image 730.Conversely, pixels have a brightness below the threshold are convertedto black on binary image 730. Converting to binary image 730 yieldsmultiple objects comprising the white portions of the image. Asmentioned previously, we are trying to recognize pedicle screws in thefluoroscope image 720. Binary image 730 yields three large white regions732, 734, and 736 with several smaller white regions. The computerincludes information related to different types of pedicle screws andcharacteristics of each type. The smaller white regions can beeliminated as not being candidates for pedicle screws.

In a step 704, an outline of the blobs are created. The computeridentifies regions of interest that could possibly be pedicle screws.Referring briefly to FIG. 33, regions of interest are shown with boxes742, 744, and 746 surrounding each blob. An outline of each blob withinboxes 742, 744, and 746 is highlighted or enhanced to show the shape ofeach blob. In one embodiment, an effective shape descriptor can be usedfor pattern recognition. One type of contour-based shape descriptor areregular moment invariants of which one set is known as a Hu momentinvariant algorithm. The Hu moment invariant algorithm is applied to theblobs to describe each object despite the location, size, or rotation.Alternatively, a Zernike Moment can be used. For example a blobdescriptor can have a particular weighted average moment of the imagepixels' intensities that represent or characterize a blob.

In a step 706, an object that contains the characteristics of a pediclescrew is placed is within the regions of interest. The object isrepresented similarly to the blob whereby the object and the blobs canbe compared. In the example, the object is the pedicle screw. In oneembodiment, the object and blobs are represented by pixels. In a step708, a least square fit (LSF) algorithm determines the error between theblob outline and the perimeter of the object (example—pedicle screw). Ina step 710, the object is moved to the next pixel then the LSF algorithmcalculates the error. The process continues where the object is movedpixel by pixel in the ROI until a minimum error is found. In a step 712,the object is then rotated until a least squares fit error minimum isfound. In a step 714, the object is then scaled until a least squaresfit error minimum is found. In a step 716, the object translation,rotation, and scale cycle is iterated through until all minimums arefinalized for each region of interest. At this point the object isfitted to the blob. The process disclosed above can be used by the spinemeasurement system 10 in assessing fluoroscope images, an image providedby optical measurement probe 12, or images provided in real-time byoptical measurement probe 12.

FIG. 34 is block diagram 800 illustrating the use of computer vision forpedicle screw or vertebra identification from a fluoroscope image or animage provided by an optical measurement probe in accordance with anexample embodiment. A method of recognizing an object using the spinemeasurement system 10 of FIG. 1, FIG. 5, or FIG. 22 is disclosed. Themethod can be practiced with more or less steps and is not limited tothe order of steps shown. The method is not limited to the spine examplebut can be used for hip, knee, shoulder, ankle, elbow, spine, hand,wrist, foot, bone, and musculoskeletal system. The components listed inthe method can be referred to herein below and are disclosed in FIG. 1,FIG. 5 or FIG. 22. In general, a fluoroscope 18 is one source forproviding an image of the spine for object recognition such as vertebraendplates or pedicle screws for measurement of angles related to spineshape. Optical measurement probe 12 is another source for providing animage or images. Images are provided to remote station 20 where computervision software is run to recognize pedicle screws, vertebra, vertebralandmarks or features, and spine landmarks or features. Block diagram800 can use the computer vision steps disclosed in FIG. 30.

In a step 802, an image is acquired from the camera (e.g. opticalmeasurement probe 12). In the example, an image is taken of monitor 26of fluoroscope 18 by optical measurement probe 12. The image of afluoroscope image is received by remote station 20 from opticalmeasurement probe 12. In a step 804, the image is filtered to reducenoise. This reduces noise introduced by system 10 such as the imagingsensor of the camera. In a step 806, a binary image is created. Thebinary image highlights regions of interest related to the objects to berecognized on the image. For example, pedicle screws, vertebralandmarks, or vertebra endplates can be targets for the computer visionrecognition. In a step 808, blobs are located and characterized. In theexample, blobs corresponding to pedicle screws, vertebra, or vertebralandmarks can be located. In a step 810, candidate blobs are determinedto be targets. In general, the candidate blobs are selected as having ahigh degree of probability of being the object to be recognized. A Hualgorithm can be executed on the candidate blobs to put them in amathematical form where they can be compared. In a step 812, the objectbeing recognized is fitted to each blob. The object has a mathematicalmodel or form where the candidate blobs and the object can be compared.It should be noted that the computer has the dimensions and shape of theobject as well as other object descriptors. The fitting process canrequire translation, rotation, or scaling of the object. The fittingprocess places the object to the blob having a minimum error. In a step814, the object's 2D spatial position is now known. In general, theamount of translation, rotation, and scaling of the object defines thespatial position. The positions relative to other recognized objects isalso known. Thus, specific objects within the image have been identifiedand their position, rotation, and scaling are also known on the imagerelative to one another.

In the pedicle screw example the computer vision program knows thatthere is a threaded region and a head region of the screw. The programfurther knows that only the threaded region is screwed into bone of thevertebra. The head region is outside the vertebra. The program can focuson the threaded region of the blob and fitting the threaded region ofthe object to the blob in this part of the region. As mentionedpreviously, the program knows the dimensions of the pedicle screw andmore specifically the threaded region of the pedicle screw. The computervision program can then recognize the vertebra and details of thevertebra (such as end plates) since the vertebra has a dimensional andspatial position relative to an installed pedicle screw. For example,the computer vision program can recognize endplates of the vertebra inrelation to the threaded region of the pedicle screw using informationgenerated in recognizing the pedicle screw and knowing where theendplates should be in relation to the pedicle screw.

FIG. 35 is block diagram 900 illustrating Cobb angle measurement inaccordance with an example embodiment. FIG. 34 discloses recognizingpedicle screws and vertebra endplates using computer vision software.Included in the recognition process is a fitting process thattranslates, rotates, and scales the object to the blob in the imagewhereby the position of the object is known. The components listed inFIG. 1, FIG. 5 or FIG. 22 can be used in the process. Similarly, stepsin FIG. 30 and FIG. 34 can be used in block diagram 900. In a step 902and disclosed in FIG. 34, the X,Y location, the rotation, and the scaleof the objects recognized in the image provided by optical measurementprobe 12 to remote station 20 is known. In one embodiment, the spine ismanipulated in real-time by a surgeon. Optical measurement probe 12provides images of pedicle screw extenders coupled to vertebra of thespine. Computer vision software is then used to measure the Cobb angleas disclosed herein below.

In a step 904, remote station 20 has stored in memory thecharacteristics of each recognized object. In general, the objects haveknown characteristics and features that are used in the computer visionprogram executed by the computer. Characteristics such as an axis of theobject or X,Y dimensions of the object are known. Similarly, a featuresuch as endplates of vertebra or landmarks on vertebra are known. Also,spacing or angles between characteristics are known such as pediclescrew to endplate angles. In the example, pedicle screws are placed invertebra of the spine. Pedicle screw extenders are coupled to thepedicle screws. The pedicle screw extenders are in the field of viewoptical measurement probe 12 of the spine measurement system. Remotestation 20 running computer vision receives image data from opticalmeasurement probe 12. The computer vision software recognizes eachpedicle screw extender. As a surgeon manipulates the spine to changespine shape, the pedicle screw extenders will move. In a step 906, asoftware algorithm is used to determine the spatial relationship amongthe field of objects recognized by the computer vision software. In astep 908, the surgeon manipulates the spine thereby moving the pediclescrew extenders. As disclosed the pedicle screw extender couples to apedicle screw. The characteristics of the pedicle screw extender areknown. The computer vision software can use the characteristics of thepedicle screws extenders to locate the heads of the pedicle screws towhich they couple and the threaded region of the pedicle screws in eachvertebra. Remote station 20 can relate the threaded region of thepedicle screw to a vertebra and a vertebra features such as theendplates of the vertebra. Thus, the movement of the pedicle screwextenders can be related to movement of the endplates using computervision. In a step 910, as the system calculates the angle betweenpedicle screw extenders the computer then calculates Cobb angle from thepedicle screw to endplate relationship. The pedicle screw to endplateangles are known and can be used with the pedicle screw extender anglesto calculate the Cobb angle.

FIG. 36 is block diagram 1000 illustrating a tracking of targets such aspedicle screw extenders in accordance with an example embodiment. FIG.35 disclosed a process of recognizing pedicle screws, vertebra, vertebraendplates, and landmarks using computer vision software. Included in therecognition process is a fitting process that translates, rotates, andscales the object to the blob in the image whereby the position of theobjects are known relative to one another. The components listed in FIG.1, FIG. 5 or FIG. 22 can be used in the process. Similarly, steps inFIG. 30 and FIG. 34 can be used in block diagram 1000.

Pedicle screw extenders are coupled to pedicle screws. Each pediclescrew is screwed into a vertebra. As mentioned previously, the pediclescrew extenders are targets that can be recognized by the computervision software executed on the computer. The pedicle screw extenderscan have 2D or 3D markings to support the measurement of position androtation of each target. In a step 1002, an image is acquired from acamera. In one embodiment, the pedicle screw extenders are in a field ofview of optical measurement probe 12. Optical measurement probe 12 iscoupled to remote station 20 for processing image data using computervision software. In a step 1004, the image is filtered to reduce noise.In one embodiment, the noise is generated by the image sensor of thecamera. In a step 1006, a binary image is created. As disclosed above, athreshold is determined that supports recognition of an object (e.g.pedicle screw extender). The pixels are converted to either black orwhite. In a step 1008, “blobs” are located and characterized withinregion of interests (ROI). In one embodiment, the blobs are white in thebinary image. In a step 1010, candidate blobs are determined to betargets. In general, blobs that do not have the characteristics of apedicle screw extender are eliminated. In a step 1012, region ofinterests are created. Each region of interest is located around a blobthat has a high probability of being the object. In the example, eachregion of interest comprises a blob corresponding to a pedicle screwextender. In a step 1014, the object is fitted to the blob. In thisprocess the pedicle screw extender is fitted to blob. The fittingprocess can include a least square fit algorithm that determines theerror between the blob outline and the perimeter of the pedicle screwextender. The pedicle screw extender is moved pixel by pixel with aleast square fit being calculated with each movement until a minimumerror is found. Similarly, the pedicle screw extender is scaled,translated, and rotated until least square fit minimums are finalized.In a step 1016, the 3D pose is derived. In general, a 3D pose estimationdetermines the transformation of the pedicle screw extender (or target)in 2D and yields the pedicle screw extender in 3D. The pose estimationcan comprise SolvePnP, SOlvePnPRANSAC, or other type of pose estimator.The 3D pose estimation or algorithm can estimate the 3D rotation andtranslation of the pedicle screw extender from a 2D image using a 3Dmodel of the pedicle screw extender. The dimensions and features of thepedicle screw extender are known. Also used in the pose algorithm arethe related points between the 2D image and the 3D model. The 3D posewill place each pedicle screw extenders in relation to each other in 3Dspace. The angle between pedicle screw extenders can be measured by thecomputer. The location of each pedicle screw can be calculated from acorresponding pedicle screw extender as the position, rotation, anddimensions of the pedicle screw extender are known. The system includespedicle screw data, pedicle screw installation data, and angle data fromprevious measurements which can also be used in the calculations.Similarly, the position of the vertebra or vertebra endplates can becalculated from the location of the pedicle screw. Thus, knowing theposition of each pedicle screw extender in 3D space determines theposition of each corresponding vertebra in relation to one another. ACobb angle and other angles/rotation can be calculated that provide ameasure of the spine shape in real-time using the computer visionsoftware. LLR measured absolute positions with respect to the cameracoordinate system.

FIG. 37 is block diagram 1100 illustrating a pose derived 3D location inspace from a 2D image in accordance with an example embodiment. Thecomponents listed in FIG. 1, FIG. 5 or FIG. 22 can be used in theprocess. The steps in FIG. 30 and FIG. 34 can be used in block diagram1100. One or more targets are in the field of view of opticalmeasurement probe 12. A camera in optical measurement probe 12 providesimages to a computer executing computer vision software. In oneembodiment, the targets have images on them. For example, the images canbe a plurality of circles where each circle has a different diameter.The dimensions of each circle are provided to the computer.

In a step 1102, a 3D object will display a distorted 2D image if theobject is rotated away from the front plane of the camera. The rotationof the object can be calculated based on the distortion of the images onthe target. In the example, rotation away from the front plane of thecamera would make the circles appear as ellipses. The shape of theellipses would correspond to the amount of rotation away from the frontplane of the camera. In a step 1104, a 3D object's image will changescale proportionally with the distance from the camera. In general,increasing a target distance from the camera will proportionally reducethe size of the images on the target as seen by the camera. Conversely,decreasing a target distance from the camera will proportionallyincrease the size of the images on the target as seen by the camera. Therelational distance of the targets can be determined using the knownscaling. In a step 1106, a 3D object rotated around the target to cameraaxis will be obvious. The position of the object is noted in relation tothe target or targets. In a step 1108, the pose algorithm will “move”the camera's perspective (virtually) until the image is an exact fit toa trained image. In one embodiment, the pose algorithm will “move” thecamera's perspective spherically until the ellipses seen on the targetbecome circles. The pose algorithm then knows the position of thetarget. Similarly, from the size of the circles we can determine thedistances. In a step 1110, the movements of the camera perspectivecreates X, Y, Z dimension and X, Y, Z rotation output yielding 6 degreesof freedom data.

FIG. 38 is block diagram 1200 illustrating a rod measurement inaccordance with an example embodiment. The components listed in FIG. 1,FIG. 5 or FIG. 22 can be used in the process. The steps in FIG. 30 andFIG. 34 can be used in block diagram 1100. A rod for the spine is placedin a field of view of optical measurement probe 12. An encoded collar isplaced on the rod. The dimensions of the encoded collar is provided tothe computer for reference. The encoded collar has markings that can beread by a computer and corresponds to an angular orientation of the rod.A camera in optical measurement probe 12 provides images to a computerexecuting computer vision software. The rod is rotated at least 180degrees. In one embodiment, the rod and encoded collar is rotated one ormore revolutions in the field of view of optical measurement probe 12.The rod can be rotated by hand or by a machine. The camera will take aplurality of images during a single revolution of the rod. For example,20-360 images can be taken during a single rotation.

In a step 1202, an image is acquired from the camera. The image isprovided to a computer executing computer vision software. In oneembodiment, the camera is coupled to the computer to provide a pluralityof pictures during the rotation. In a step 1204, the image is filteredto reduce noise. In one embodiment, noise from the imaging sensor isremoved. In a step 1206, a binary image is created. The binary imagewill include the encoded collar. In a step 1208, “blobs” are located andcharacterized within regions of interest. In a step 1210, candidateblobs are determined to be an encoded collar. In a step 1212, a regionof interest is created based on scale. In a step 1214, the rod isidentified. In a step 1216, a 2D measurement is made. The process isrepeated for each image taken by the camera. Further detail on measuringthe rod can be found in FIG. 29.

It should be noted that very little data exists on implanted orthopedicdevices. Most of the data is empirically obtained by analyzingorthopedic devices that have been used in a human subject or simulateduse. Wear patterns, material issues, and failure mechanisms are studied.Although, information can be garnered through this type of study it doesyield substantive data about the initial installation, post-operativeuse, and long term use from a measurement perspective. Just as eachperson is different, each device installation is different havingvariations in initial loading, balance, and alignment. Having measureddata and using the data to install an orthopedic device will greatlyincrease the consistency of the implant procedure thereby reducingrework and maximizing the life of the device. In at least one exemplaryembodiment, the measured data can be collected to a database where itcan be stored and analyzed. For example, once a relevant sample of themeasured data is collected, it can be used to define optimal initialmeasured settings, geometries, and alignments for maximizing a patient'sQuality Of Life or the life and usability of an implanted orthopedicdevice.

FIG. 39 is block diagram 1400 of an automated orthopedic process usingone or more fluoroscope images to locate musculoskeletal structures ororthopedic devices and the position of each structure or device relativeto one another to generate quantitative measurement data in accordancewith an example embodiment. The order of the blocks is not fixed and canbe placed in any sequence. In general, the automated orthopedic processuses computer vision to identify musculoskeletal structures ororthopedic device from one or more fluoroscope images. In the oneembodiment, a weighted average moment is applied to the image todescribe an object. The image moment can correspond to pixel intensityor the change in intensity of the pixel intensity. The image moment isused to identify objects in the image. The system supportsidentification of musculoskeletal structures or orthopedic devices eventhough the shapes and sizes of a musculoskeletal can vary throughout thepopulation. In one embodiment, data related to the orthopedic devices isprovided to the computer and computer vision software to supportidentifying the object in an image. Examples of an orthopedic device areprosthetic components, screws, or other devices that are temporarily orpermanently coupled to the musculoskeletal system.

A system used in orthopedic surgery is shown in FIG. 1 and describedherein above. For example, the system can be used to identify amusculoskeletal structure or an orthopedic device from a fluoroscopeimage. The system can produce quantitative measurement data that can beduring surgery. In one embodiment, the measurement data can relate toalignment or position of structures or devices in the musculoskeletalsystem. In the example disclosed herein, vertebrae of the spine can beidentified, the position of each vertebra in relation to other vertebracan be measured, and a measurement such as spine curvature can bemeasured. Furthermore, orthopedic devices such as pedicle screws can beidentified in the fluoroscope image along with each vertebra. Thepositional relationship between a pedicle screw and a vertebra can beidentified and the positional relationship can be used in otherapplications as the position relative to one another is permanent. Thus,knowing a location of the pedicle screw also determines the location ofthe vertebra to which it couples.

The automated orthopedic process 1400 can use the components ofmeasurement system 10 of FIG. 1 to identify musculoskeletal structuresand/or orthopedic devices. Orthopedic process 1400 can be used for anymusculoskeletal structure or orthopedic device to be identified within afluoroscope image. An example is provided in process 1400 where themusculoskeletal structures are vertebra and the devices are pediclescrews that are screwed into vertebra. In a block 1402, inputs areprovided to the measurement system. In general, the measurement systemincludes a computer. The computer includes software, memory, and adisplay to support orthopedic process 1400. The computer can receive andprocess data, run programs, perform calculations, and apply a computervision program that acquires, processes, and analyzes images related tothe musculoskeletal system for use in a medical or surgical environmentin real-time. The input to the system comprises one or more images ofthe musculoskeletal structures and/or devices. In the example, the imageis a fluoroscope image. Examples of other image types are MRI, CT,Ultrasound, and X-Ray to name a few types of images that can be providedto measurement system 10. The fluoroscope image can be directly portedto the computer from the fluoroscope or a camera in the orthopedicmeasurement system can be used to take an image of the fluoroscope imagethat is provided to the computer. In either case, one or morefluoroscope images are provided to the computer. In the example thefluoroscope image is an image of vertebrae of the spine. The fluoroscopeimage can further include pedicle screws that have been placed into thevertebrae. Image moments are provided to the measurement system. In theexample, the image moments can describe pedicle screws. In oneembodiment, Hu moments are used to describe the pedicle screws. The Humoments can be used in conjunction with other image moments. Theadvantage of using Hu moments is that they can be invariant undertranslation, scale, and rotation. Alternatively other image momentscould be used. The computer will also have data and information on thetype of pedicle screw being used. The data and information includes thedimensions and shape of a pedicle screw being used as well as otherdescriptive metrics that can be used in process 1400.

In a block 1404, features are found in the fluoroscope image. The block1404 receives inputs from block 1402. In general, the image moments areused to locate regions in the fluoroscope image having similarcharacteristics. In the example, the computer and computer visionsoftware identifies regions where the Hu moments of the pedicle screwscorrespond to what is seen in the fluoroscope image. Part of what isdone is by the computer vision software is fitting by translation,rotation, and scaling to fit the pedicle screw image. In a block 1406,the musculoskeletal structure or device is superimposed in the regionsfound in block 1404 corresponding to the image moments. In the example,optimal locations of pedicle screws are identified vis-à-vis fluoroscopeimage Hu moment regions. The pedicle screw image can then besuperimposed by the computer and computer vision software at theidentified locations. Thus, the pedicle screws have been identified inthe image and that can be corroborated by a user of the system fromlooking at the superimposed musculoskeletal structures or orthopedicdevices. In the orthopedic device (pedicle screw) is fastened to amusculoskeletal structure (vertebra). The musculoskeletal structure islocated in the next step. In a block 1408, a feature of themusculoskeletal system is identified. In the example, a vertebra edge isfound. The vertebra edge in the fluoroscope image corresponds to avertebra end plate. The vertebral plate is located by the computer andcomputer vision software on the fluoroscope image. A vertebra is boundedby two endplates. In one embodiment, the vertebral plate is parallel tothe long axis of the optimally placed pedicle screw. The relationshipbetween a vertebra and pedicle screw(s) is now known by the computer andcomputer vision software. In one embodiment, a user can view and approvethat the computer vision software has identified the components ofinterests.

FIG. 40 is a block diagram 1500 illustrating steps involved with acomputer and computer vision software to identify musculoskeletalstructures or devices in one or more images provided to the computer inaccordance with an example embodiment. Block diagram 1500 provides moredetail corresponding to the block diagram 1404 of FIG. 39. In general,block diagram 1500 illustrates steps to find a musculoskeletal structureor an orthopedic device in an image. For example, system 10 of FIG. 1includes computer vision software that can find a feature in an imagecaptured by system 10. The order of the blocks are not fixed and can beperformed in any sequence. In the example, one or more fluoroscopeimages are provided to the computer. In block 1502, an image received bythe computer is converted to grayscale. In one embodiment, the image isa digital image. In the example, the image received is a digitalfluoroscope image. In general, grayscale is a range of shades of gray.Within the grayscale the lightest possible shade is white while thedarkest shade is black. The converted image can include wide variationscorresponding to different grayscale shades. In block 1504, a filter isapplied to the grayscale image generated in block 1502. Morphologicalimage processing comprises non-linear operations related to the shape offeatures in the image. Morphological operations can comprise erosion,dilution, Gaussian filtering, and Laplacian filtering. In oneembodiment, a Gaussian filter is applied to the grayscale image.Alternatively, other filter types could be used on the grayscale image.A Gaussian filter removes high frequency components or noise from animage and corresponds to a low pass filter. Furthermore, the Gaussianfilter can soften hard edges of the image. The impulse response of aGaussian filter is a Gaussian function. In one embodiment, an imagealgorithm to enhance a grayscale image comprises subtracting a blurredversion of an original grayscale image from another, less blurredversion of the original grayscale image. The blurred images are formedby convolution. For example, two blurred grayscale images can be createdby convolving the original grayscale image with Gaussian kernelscomprising different standard deviations. Gaussian kernals can also beused to quantify the image so features, size, rotation, etc. of amusculoskeletal structure or device can be detected.

In block 1506, equalization is applied to the filtered grayscale imageof 1504. Equalization can be used to improve the contrast in images. Inone embodiment, an adaptive histogram is applied to the filtered imageof the musculoskeletal structures or the orthopedic devices.Alternatively, other equalization could be used. In block 2104,equalization is applied to the grayscale image of block 2102. Theequalization is applied to improve contrast on the grayscale image tosupport identification of the musculoskeletal structures or orthopedicdevices. A histogram is created of the entire image's intensity value,then a scaling factor is applied to each bin so that the distributionacross the intensity range matches a pre-determined profile. The binscaling factor is then applied to each corresponding pixel intensity inthe image thereby normalizing the image contrast. For example, someimages might have high peaks clustered around a value of 90-120, thealgorithm will increase 0-89 and 121-255 and reduce 90-120. In block1508, an image threshold is found or identified after equalization wasapplied in block 1506. The image threshold corresponds to a point wherepixels having a brightness above the threshold are converted to whiteand all pixels below the threshold are converted to black. The imagethreshold is used to support identification or locating amusculoskeletal structure or orthopedic device in an image. Imagethresholding can be binary, gray scale, adaptive histogram, or othertype. In the example, the threshold is selected to support locatingpedicle screws in the image. In block 1510, the grayscale image isconverted to a black and white image after selecting the threshold inblock 1508. The black and white image can also be called a binary image.In one embodiment, the musculoskeletal structure or orthopedic device inthe binary image is in white. In the example, the pedicle screws imagesare white in the binary image. In block 1512, an edge detector algorithmis applied to the black and white image created in block 1510. In oneembodiment, a Canny edge detection algorithm is used on the black andwhite image. Alternatively, Sobel, Roberts, Prewitt, Laplacian, or otheredge detection algorithms can be used. The edge detection algorithmcharacterizes boundaries that support identification of themusculoskeletal structure or device. In general, the edges correspond toareas of high contrast. In the example, the pedicle screw will havescrew threads for cutting into bone and holding the screw in place. Thescrew threads are a high contrast region where the pixel intensity canvary significantly over a small range of the image. The Canny edgedetection algorithm has low error rate in detecting edges. Moreover, theCanny edge detection algorithm does not detect non-edges, identifiesedges that are localized, and has one response for a single edge.Referring briefly to FIG. 47, an edge detection algorithm is applied toa black and white image to yield a Canny edge image 2200. The Canny edgeimage 2200 contains the binary image of the pedicle screws. Pediclescrews 2202, 2206, and 2208 are shown displaying the edges identified bythe edge detection algorithm.

Further processing is performed by the computer and computer visionsoftware to support finding features of the musculoskeletal structure ororthopedic device. In block 1514, small objects are removed from theedge detected black and white image of block 1512. After the edgedetection algorithm has been run on the binary image there may be smallobjects identified in the image that cannot possibly be themusculoskeletal structures or orthopedic devices. As mentioned, theobjects being identified are pedicle screws. In one embodiment, objectsare removed from the image that do not have the size or shape of apedicle screw. In block 1516, the objects are dilated and the edges areclosed after the small objects have been removed in block 1514. Ingeneral, applying a dilation process or algorithm to the identifiedobjects in the image will “grow” the feature. For example, after runningan edge detector algorithm and applying dilation, the edges of theobject(s) can thicken. Dilation can also close the gaps between edgesthat are close to one another. To illustrate further, a region of apredetermined size can be centered on a pixel. The dilation algorithmwould make the pixel white if a white pixel is found anywhere within theregion thereby “growing” or “filling” the feature. In the pedicle screwexample, the outline or edge of the screws in the image can be madecontiguous by the dilation algorithm. In block 1518, a fill closedcontours process or algorithm is used to fill in continuously closedfeatures after applying dilation in block 1516. In general, “blobs” areformed corresponding to the musculoskeletal structure or orthopedicdevices found in the image. In the example, a “blob” corresponds to apedicle screw that is found in the image. The fill closed contoursprocess can eliminate any nested closed contours to make a singleentity. In block 1520, the contour boundaries are found after the closedcontours are filled in block 1518. This process or algorithm creates asingle entity outline from what could have been multiple entities nestedwithin each other. In block 1522, loop processing is performed onboundaries in the image. In general, the musculoskeletal structure ororthopedic device in the image is identified as a single entity, a“blob” is created on the single entity, and a transform is applied andmatches the result with the template values. The computer and computervision program when looping on boundaries can include filling regionsclosed by boundaries, compute image moments for regions, compute L2 normof image moment of the region, compute image moment of musculoskeletalstructure or device, save in memory the boundary information, and savein memory the image moment for the region. In the example, each of theidentified pedicle screws comprise a single entity, a “blob” is createdof each pedicle screw, a Hu transform is applied to each pedicle screwwherein the result of the Hu transform matches the template values ofthe pedicle screw. In block 1524, an output is provided by the computerand computer vision software. The output can comprise the X and Ylocation of the centroid of the identified musculoskeletal structure ordevice and the rotation and scale of the musculoskeletal structure ordevice. The computer and computer vision software then places a UIobject on top of the captured image. The output can further include animage converted to a binary image, a boundary list, boundary labels,image moments for all regions, the number of found musculoskeletalstructures or devices, and image moments for found musculoskeletalstructures or devices. The user can then agree or disagree that theidentified musculoskeletal structures or orthopedic devices have beencorrectly identified.

FIG. 41 is a block diagram 1600 illustrating steps involved with acomputer and computer vision software to place and identifymusculoskeletal structures or orthopedic devices in one or more imagesprovided to the computer in accordance with an example embodiment. Blockdiagram 1600 provides more detail corresponding to the block 1406 ofFIG. 39. In general, block diagram 1600 illustrates steps to place amusculoskeletal structure or an orthopedic device that has beenidentified overlying an image. For example, system 10 of FIG. 1 includescomputer vision software that can place the musculoskeletal structure ororthopedic device overlying the image. In an operating room environmentoverlaying the found object allows a user of the system to determine ifthe one or more “found” objects have been correctly identified. The usercould then respond that the found objects are correct or identify errorsthat can then cause the system to correct the found objects. The orderof the blocks are not fixed and can be performed in any sequence. In theexample, one or more fluoroscope images are provided to the computer. Inblock 1602, an overlay image is created. The overlay image is areference musculoskeletal structure or a reference orthopedic device. Inthe example, the object that the system has identified in the image arepedicle screws that are attached to vertebra of the spine. An overlayimage of a reference pedicle screw is created in block 1602. In block1604, a binary image of the reference musculoskeletal or referenceorthopedic device image from block 1602 is created by the computervision software. In the example, the binary image is a pedicle screwwhich is an orthopedic device. In block 1606, the location of themusculoskeletal structure or orthopedic device is calculated. In theexample, the location of the pedicle screws are calculated. The locationof the pedicle screws can be found using an image moment. In oneembodiment, the zero and first Hu moment is used to determine thecentroid location for each for each pedicle screw. Thus, the centroid ofeach pedicle screw corresponds to the location. In block 1608, the scalefactor for each musculoskeletal structure or device that has beenidentified in an image is calculated after block 1606. The scale factoris calculated by using the Hu Moments to determine the area of thepedicle screw blob. In the example, the scale factor is calculated forthe pedicle screws. The scale factor can be calculated relative to thecentroid of each pedicle screw. This found scale factor is then appliedto the placed reference pedicle screw. In block 1610, the rotation ofthe musculoskeletal structure or orthopedic device can be calculatedafter block 1608. In general, the image of the musculoskeletal structureor orthopedic device is rotated for best fit. In the example, thecalculation provides the rotation of the pedicle screw image for bestfit. In one embodiment, the rotation of each pedicle screw is calculatedusing a L2 Norm to achieve a best fit. In block 1612, a superimposedimage is displayed overlying the original image after block 1610. Theimage of the identified feature has been located, scaled, and rotated.Placing the image overlying the original image provided to the systemshould place the identified objects overlying the musculoskeletalstructures or orthopedic devices on the original image. In the example,the original image is a fluoroscope image. The overlay image includesthe identified musculoskeletal structures or orthopedic devices. In theexample, the identified pedicles screws by the computer vision softwareare overlayed on the image. The overlay image is placed overlying thefluoroscope image on a display and the computer vision identifiedpedicle screws should align with the pedicle screws in the fluoroscopeimage. In one embodiment, the user can indicate that the computer andcomputer vision software has correctly identified the features ofinterest. Alternatively, the user can indicate that the features ofinterest are incorrect and that the fluoroscope image needs to bereassessed.

FIG. 42A is a block diagram 2100 illustrating steps involved with acomputer and computer vision software to place and identifymusculoskeletal structures or orthopedic devices in one or more imagesprovided to the computer in accordance with an example embodiment. Inthe example, musculoskeletal structures are identified by the computerand computer vision software from one or more images. As disclosedherein above, pedicle screws are placed in vertebra of the spine. Thepedicle screw is screwed into a vertebra at a predetermined angle andpositioned between the endplates of the vertebra thereby determining arelationship between pedicle screw and the vertebra. Typically, the headof the pedicle screw will couple to a surface of the vertebra. Afluoroscope image is used by the computer and computer vision softwareto locate the pedicle screws as disclosed herein above. In block diagram2100, a process for identifying vertebra is disclosed. Morespecifically, the endplates of the vertebra in the fluoroscope image areidentified. The endplates can be used to define each vertebra as theyform a boundary between vertebrae. The endplates will look like a lineor edge on the fluoroscope image. Thus, the pedicle screws and thecorresponding vertebra in which they are installed are identified withinthe fluoroscope image. Moreover, the position of each pedicle screw isknown in relation to a corresponding vertebra. This relationship can bemeasured from the fluoroscope image and the measurements can be used tocalculate relative positions of vertebra and pedicle screws to oneanother. For example, a Cobb angle, dimensional position, spinecurvature, listhesis can be calculated as disclosed herein above andmeasurement data related to position, rotation, translation, and anglerelated to musculoskeletal structures or orthopedic devices. Also, thepositional relationship between the pedicle screws and the vertebra canbe used to track spine position as it manipulated or modified duringsurgery as disclosed herein above. In other words, knowing the positionof the pedicle screw also determines the position of the correspondingvertebra to which it couples.

Block diagram 2100 provides more detail corresponding to the block 1408of FIG. 39. In one embodiment, block diagram 2100 illustrates steps toidentify endplates of vertebra on the fluoroscope image. The system 10of FIG. 1 can be used to identify musculoskeletal structures andorthopedic devices such as endplates of vertebra. The order of theblocks of block diagram 2100 are not fixed and can be performed in anysequence. In the example, one or more fluoroscope images are provided tothe computer. In block 2102, a grayscale image of the fluoroscope imageis created. In block 2104, equalization is applied to the grayscaleimage of block 2102. The equalization is applied to improve contrast onthe grayscale image to support identification of the musculoskeletalstructures or orthopedic devices. In one embodiment, an adaptivehistogram is applied to the grayscale image of the spine. The adaptivehistogram applies several histograms, where each histogram is applied toa different region of the image. The adaptive histogram willredistribute light and dark areas to improve the contrast of the imagein a region. In block 2106, an edge detection routine is applied to thegrayscale image that was equalized. The edge detection routineidentifies points in a grayscale image where discontinuities exist. Forexample, an edge detection routine can identify where brightness changessignificantly. The change in brightness typically indicates an edge inthe image. In one embodiment, Canny edge detection is used on thegrayscale image. The edge detection is further enhanced by setting lowand high thresholds for the Canny edge detection routine that furtherhighlights changes in brightness. In block 2108, a differential of anedge detection is employed to mask out a feature. In the example adifferential of the Canny edges are found. Cannydiff (CannyL-CannyH) isused to mask out the musculoskeletal structures or orthopedic devices.In the example, the pedicle screws are masked out to simplify the taskof identifying the endplates. As disclosed above, the pedicle screwswere previously identified by the computer vision software and removingthe pedicle screws simplifies endplate detection. In block 2110, theimage is cleaned up by the computer and computer vision software. Thecleanup can comprise removing pixel regions below a predetermined size.In one embodiment, pixel regions smaller than 10×10 are removed from theimage.

In block 2112, regions where musculoskeletal region or orthopedicdevices have been found are looped on in the computer vision software torepeat a flow disclosed herein below. In the example, pedicle screwshave been found previously in the fluoroscope image provided to thecomputer. It is known that a pedicles screw is inserted into a vertebraof the spine. Thus, focusing on a region where the pedicle screw hasbeen located will include endplates of the vertebra. The loop as statedcomprises j=1 to NScrew where NScrew is the number of identified screwregions. Referring to FIG. 47, pedicle screws 2202, 2206, and 2208 werefound. Thus, NScrew would be 3 (e.g. 2202, 2206, 2208) for FIG. 47. Eachpedicle screw is coupled to a different vertebra. The endplates can befound in regions around pedicle screws 2202, 2206, and 2208 in thatimage. In block 2114, a feature extraction algorithm is applied to theimage. In the example, the feature extraction algorithm can be selectedfor identifying the endplate of a vertebra. The endplates on thevertebra will appear as lines on the image. In one embodiment, a Houghtransform is used. Alternatively, the shaped based feature extractioncan also comprise template matching, fuzzy Hough transform, blobextraction, or other extraction type. The Hough transform can be used tofind features such as lines, curves, or other features that may bedefined in a parametric form. In one embodiment, the Hough transformangle range is set up for region J to support identification of theendplates. In block 2116, the feature extraction algorithm iscalculated. In the example, the Hough transform is computed to find thefeature. The Hough transform can identify lines in the image. In oneembodiment, lines are selected that are + or −10 degrees from a pediclescrew axis of the Canny diff image for the set up angle range. In block2118, a predetermined number of features are selected. In the example,endplates of vertebra are being identified by the computer visionsoftware on the image. In one embodiment, the 50 highest ranking Houghpeaks are found. The Hough peaks correspond to the parameters of modelsto be detected (e.g. vertebra endplates). The higher ranking peaks havea higher probability of being endplates or lines corresponding toendplates of a vertebra. In block 2120, the feature extraction processfurther eliminates identified features to reduce the pool of identifiedfeatures on the image. In the example, lines corresponding to endplatesidentified by the Hough transform that are less than a predeterminedlength are eliminated. In other words, if the identified lines are lessthan the predetermined length it is likely that they are not endplatesof a vertebra. In one embodiment, Hough lines that are at least 30pixels in length are selected from the lines having the 50 highestranking Hough peaks.

In block 2122, the computer vision software loops on the each objectidentified by the extraction algorithm. In the example, linescorresponding to endplates are identified by the extraction software. Inblocks 2118 and 2120 the number of identified lines by the Houghtransform is reduced. In one embodiment, the system loops through theidentified Hough lines from k=1 to ILines where ILines is the number ofidentified lines that are at least 30 pixels in length (block 2120). Inblock 2124, a line is projected from the previously identified featurecenter of mass. In the example, the previously identified feature is apedicle screw. In one embodiment, the line is projected from the centerof mass of the pedicle screw normal to the screw angle until itintersects Hough line k. FIG. 42B is a continuation of block diagram2100 in accordance with an example embodiment. In block 2130, thecomputer vision software finds the shortest line segment greater than apredetermined value (Mindist) from identified features 1 to ILines. Inblock 2132, the identified shortest line segment is then overlayed ordisplayed on the image. The identified shortest line segment isoverlayed on the image and can be stored as the identified feature. Inthe example, the identified shortest line segment corresponds to avertebra endplate. The line is overlayed on the fluoroscope image. Inone embodiment, the user of the system can acknowledge that the computerand computer vision software has correctly identified the vertebraendplate. The system and computer vision software can also identify thespecific vertebra and label the vertebra on the image. The linecorresponding to the vertebra endplate is stored in memory coupled tothe computer and system. In block 2136, the routine loops back to block2112 where another region is selected corresponding to a previouslyidentified object. In the example, the process will loop back on eachregion having a pedicle screw.

FIG. 43 is a block diagram 1800 illustrating tracking of one or moreobjects using computer vision software in real-time in accordance withan example embodiment. In general, system 10 of FIG. 1 disclosed hereincan be used for tracking one or more musculoskeletal structures ororthopedic devices. System 10 includes a camera that provides video tothe computer. The computer includes computer vision software foridentifying and tracking the one or more objects in real-time. In theexample disclosed above, pedicle screws are coupled to vertebra of thespine during spine surgery. Pedicle screw extenders are coupled to thepedicle screws. As their name implies, the pedicle screw extendersextend from the spine in a manner where they are visible to a surgeon.The pedicle screw extenders are also visible to the camera providingreal-time video to the computer. In general, the computer and computervision software can identify and track the pedicle screw extenders inreal-time in 3D space. The computer vision software can provide positionof the pedicle screw extenders relative to one another. As the surgeonmanipulates the spine the pedicle screw extenders change positionrelative to the vertebra to which it couples. Block diagram 1800illustrates steps to capture the movement of the pedicle screwextenders. The computer vision software provides position data of eachpedicle screw extender. In one embodiment, a pedicle screw extender iscoupled to a pedicle screw such that the pedicle screw extender isrigidly attached and maintains a fixed geometric position in relation tothe pedicle screw. Thus, as disclosed herein above, the position of thepedicle screw extender corresponds to the position of the pedicle screw.Similarly, the position of the pedicle screw corresponds to the positionof the vertebra to which it couples. The measurement data related to thepedicle screws and vertebrae positions in 3D space have been captured asdisclosed herein above from the fluoroscope image or images. Thecomputer and computer vision software links the tracking of the pediclescrew extenders by the camera to positions of the vertebrae. Thereal-time measurement data can be used with the measurement data fromthe fluoroscope image to support and provide quantitative measurementdata related to position, rotation, translation, and angle relatedmusculoskeletal structures or orthopedic device or relationalmeasurement data such as dimensional positioning, curvature, orlisthesis from the measurement data.

In block 1802, video is provided to the computer having computer visionsoftware. The objects being monitored are within the video frame of thecamera. The objects are being monitored in real-time comprisemusculoskeletal structures or orthopedic devices. In the example, theobjects being monitored are pedicle screw extenders. As disclosed above,the pedicle screw extenders are coupled to pedicle screws that have beenfastened to vertebra of the spine. The pedicle screw extenders can betracked with a complete or partial view as long as each pedicle screwextender can be identified within the video frame by the computer visionsoftware. In block 1804, an object detection algorithm is employed totrack musculoskeletal structures or orthopedic devices in real-time. Inone embodiment, a cascade detector is trained to detect an object.Alternatively, a Voila-Jones Detector, SVM (Scalar Vector Machine), Bagof Features (Bag of Words) or other object detector can be used. Cascadedetectors perform object detection in an efficient manner. A Cascadedetector has sequential stages that are designed to cull out objects.Typically, each stage of a Cascade detector becomes progressively morecomplex where each stage eliminates negative images while leavingpositive images that have a high probability of being the object ofinterest. Ideally, the final stage of the Cascade detector will yieldthe objects of interest. In the example, the objects are pedicle screwextenders. In one embodiment, the Cascade detector identifies a numberof pedicle screw extenders in a video frame and frames a Region ofInterest (ROI) around each pedicle screw extender. In block 1806, themusculoskeletal structures or orthopedic devices are located in space.In general, the translation and angle of each musculoskeletal structureor orthopedic device is estimated with respect to a camera sensor plane.In the example, the pedicle screw extenders are located in space. Inblock 1808, one or more filters are applied to the video frame orobjects. In the example, the pedicle screw extenders estimated positionsare smoothed using a filter. In one embodiment, the filter is a Kalmanfilter or a rolling average filter. The Kalman filter is a recursivefilter that can process information as it arrives. It is an estimatorthat infers parameters of interest from indirect, inaccurate anduncertain observations, can be used in real-time applications, andprovides the estimate if the data is noisy.

FIG. 44 is a block diagram 1700 having further detail illustratingtracking of one or more identified objects using computer visionsoftware in real-time in accordance with an example embodiment. In block1702, a camera provides video in real-time to the computer havingcomputer vision software. The computer will have stored in memory thephysical characteristics of the musculoskeletal structures or orthopedicdevices being tracked. As mentioned previously, the objects of interestwill be partially or completely displayed in each video frame. Thecomputer vision software can identify an object even if a portion of theobject is obscured from view. In block 1704, a region of interest (ROI)is formed around an object. Multiple regions of interest are created ina video frame when multiple objects are identified. In the example, aregion of interest is formed around each pedicle screw extender. Adetector is used to identify each pedicle screw extender in each videoframe. In one embodiment, the detector is a Cascade detector. Typically,the pedicle screw extenders will not move significantly from video frameto video frame. In one embodiment, once the regions of interest havebeen identified there may not be need to identify the objects and createregions of interest around the objects as new video data is provided.The movement of an object such as a pedicle screw extender within aregion of interest can be tracked and the region of interest re-centeredaround the object subsequently. Thus, the identification process is notrequired with each video frame thereby saving on the computation timerequired to track objects. In block 1706, descriptors are generated andfeatures are extracted using a feature detector. The features are a setof distinctive keypoints that can be identified in different images,viewpoints, and under noisy conditions. In one embodiment, an invariantfeature detector is used. Feature extraction reduces the amount of datathat has to be processed. Often much of the data is redundant orirrelevant. Feature extraction transforms the data into a set offeatures. The feature extraction can be used to compare identifiedpoints of interest to other points of interest in an image. Descriptorsare generated from each region of interest. The descriptors aredescriptions of visual features in images. Descriptors can also describecharacteristics such as shape, color or other descriptive characteristicthat can be used for identification. In the example, features areextracted related to a pedicle screw extender and descriptors generatedon the pedicle screw extender. In block 1708, the object location isdetermined in 3D space. In one embodiment, a solvePnP function is usedby the computer and computer vision software to determine objectlocation. SolvePnP estimates an object pose given a set of objectpoints, their corresponding image projections, the camera matrix, anddistortion coefficients. The function typically requires a set of 2D(Dimension) and 3D (Dimension) correspondences related to the camera andan image. In general, the function provides the rotation and translationof the camera with respect to the object of interest. In the example,the objects are the pedicle screw extenders. In other words, the solvePnP function creates translation and rotation vectors related to thecamera and objects.

FIG. 45 is a block diagram 1900 having further detail illustrating imagetraining using computer vision software in real-time to track one ormore objects in accordance with an example embodiment. In block 1904, acamera provides video in real-time to the computer having computervision software. In block 1902, training images are provided to supportidentification of a musculoskeletal structure or orthopedic device. Ingeneral, the training images comprise positive and negative images. Thepositive and negative images are static images purposefully gatheredprior to running the computer vision algorithm sent through a trainingalgorithm whereby a statistical model is created. The statistical modelis then used as the basis to determine if the object was found (e.g.SVM—Support Vector Machine) or in the case of a Cascade classifier usedto create the filters as disclosed herein below. The computer will havestored in memory the physical characteristics of the musculoskeletalstructures or orthopedic devices being tracked. A Cascade detectorutilizes a sequential process to eliminate negative samples and minimizefalse positive samples. The detector can be trained for differentorientations of an object. A trained classifier is comprised of a numberof weak stages. The weak stages can be decision stumps. The stages canbe trained to improve accuracy by taking a weighted average of thedecisions made by the weak stages. The classifier will provide apositive or negative response when viewing a region of an image. Apositive response corresponds to the likelihood the object can be foundand a negative response corresponds to the likelihood no object will befound. Any positive responses will be provided to the next stage. Thedetector comprises sequential stages that eliminates negatives samplesuntil only true positives remain. Ideally, each stage will have lowfalse positives if trained correctly. In general, the image training isfor a musculoskeletal structure or an orthopedic device. In the example,the image training is for a pedicle screw extender. The positive andnegative training images which yield a statistical representation of apedicle screw extender image is provided to the detector of block 1906.In one embodiment, the detector of block 1906 is a Cascade detector. Inblock 1904, a video frame is provided to the detector of block 1906. Thedetector processes the video frame to identify the number and locationof pedicle screw extenders in the frame. In block 1908, an output isprovided from the detector. The detector identifies a number of pediclescrew extender found in the video frame and provides a region ofinterest around each pedicle screw extender.

FIG. 46 is a block diagram 2000 showing steps to acquire a position of amusculoskeletal structure or orthopedic device in a video frame inreal-time in accordance with an example embodiment. As mentionedpreviously, system 10 of FIG. 1 can track the musculoskeletal structureor orthopedic device in an operating room in real-time to supportsurgery. System 10 includes a computer and computer vision software. Thepedicle screw extender couples to a pedicle screw. The pedicle screw isscrewed into a vertebra of the spine. Tracking the pedicle screwextenders provides quantitative measurement information related to thepedicle screws and vertebra. The tracking data is used in conjunctionwith quantitative measurement data generated from the computerprocessing one or more fluoroscope images using the computer visionsoftware. In the example, the surgery is spine surgery where pediclescrew extenders are tracked in real-time to generate quantitativemeasurement data related to the position of each vertebra and the shapeof the spine.

In block 2022, a camera provides a video frame. In block 2016, the videoframe is looped on to find a number of objects in the video frame. Theobjects are musculoskeletal structures or orthopedic devices. In theexample, the objects are pedicle screw extenders. In block 2002, adecision block continues to process the video frame if it is believedthat there may be other musculoskeletal structures or orthopedic devicesthat have not been found in the video frame. In block 2004, the trackingpoints are adjusted. For example, two consecutive video frames can bereviewed using a feature tracking algorithm. The feature trackingalgorithm can determine how points being tracked move from frame toframe. In one embodiment, the movement can be performed using a leastsquares fit. The adjustment includes computing new regions of interest(ROI) around the musculoskeletal structures or orthopedic devices. Inthe example, the feature tracking algorithm looks for point movementrelated to the pedicle screw extenders. The feature tracking algorithmcan be a KLT (Kanade-Lucas-Tomasi) feature tracking algorithm that workswell for real-time tracking. In block 2006, a features detector is usedon an image in a ROI from block 2004. The features detector can be usedfor object recognition, object registration, object classification, or 3dimensional (3D) reconstruction. In one embodiment, a speed up robustfeatures (SURF) detector is used on an image in the ROI. Alternatively,SIFT (scale invariant feature transformation), MSURF, ORB, FAST,(Histogram of Gradients), Brisk, Harris, or other feature detectors orfeature matching can be used. The SURF algorithm can be used for featureextraction and continuous recognition in video. The SURF algorithm cantrack objects by interest point matching and updating. The SURF detectoris used to compute SURF corner points for an image in each ROI. Ingeneral, a corner is an intersection of two edges. The cornercorresponds to a change in the gradient in the image. Both edges of acorner change directions in the image at the corner point whereby thechange can be detected readily. Block 2020 comprises the musculoskeletalstructure or orthopedic device training image data set. The trainingimage data set can include SURF points, SURF corner points, and X, Y, Zposition of each SURF point and SURF corner points. In the example,block 2020 can provide SURF points, SURF corner points, and X, Y, Zposition of each SURF point and SURF corner point of the pedicle screwextender. In block 2008, the SURF points from the image in the ROI arecompared to the SURF points of the musculoskeletal structure ororthopedic device training image. The comparison can also be betweenSURF corner points from the image in the ROI compared to the SURF cornerpoints of the musculoskeletal structure or orthopedic device. In theexample, the SURF corner points from ROI are compared to SURF cornerpoints of the pedicle screw extender image. Block 2008 receives thetraining image set from block 2020. In block 2010, the X, Y, Z, geometryis extracted for each matched SURF point or corner point. Block 2018comprises camera calibration coefficients, camera matrix, and lensdistortion parameters. In block 2012, a pose estimate is performed ofthe musculoskeletal structure or orthopedic device. Block 2018 providesthe camera calibration coefficients, camera matrix, and lens distortionparameters to block 2012. The pose estimator estimates the 3 dimensional(3D) rotation and translation of a 3D object from a 2 dimensional (2D)image. In general, the 3D pose is estimated from X, Y, Z model pointsand the 2D image points. In the example, the pose estimate is performedon the pedicle screw extender in the ROI. In one embodiment the poseestimate is performed using solvePnP (solve pose and position). In block2014, the translation and rotation estimate of the musculoskeletalstructure or orthopedic device is stored in memory coupled to thecomputer. The translation and rotation estimate is linked tomusculoskeletal structure or orthopedic device in the ROI. In theexample, the translation and rotation estimate is stored and linked topedicle screw extender in the ROI. Block 2014 then loops back todecision block 2002. The program loops through blocks 2004, 2006, 2008,2010, 2012, and 2014 until there are no more musculoskeletal structuresor orthopedic devices. In the example, the loop ends when there are nomore pedicle screw extenders in the image or regions of interest and thetranslation and rotation of each identified pedicle screw extender isdefined. Upon completing the loop, the output is filtered and displayedin block 2024. Thus, the musculoskeletal structure or orthopedic deviceis tracked with each video frame and the position in 3D space is knownin each video frame. As mentioned previously, the quantitativemeasurement data generated in this process can be used to generateparameters needed to assess or measure the musculoskeletal structure ofinterest. In the example, the pedicle screw extender 3D position can beused to calculate a Cobb angle, spine curvature, or vertebra positionand rotation as the position of the pedicle screw extender directlyrelates to the position of the vertebra to which it couples.

While the present invention has been described with reference toparticular embodiments, those skilled in the art will recognize thatmany changes may be made thereto without departing from the spirit andscope of the present invention. Each of these embodiments and obviousvariations thereof is contemplated as falling within the spirit andscope of the invention.

What is claimed is:
 1. A method of measuring musculoskeletal systemduring surgery comprising the steps of: taking one or more views of themusculoskeletal system with a fluoroscope; capturing one or morefluoroscope images with a camera wherein the camera is configured tosend the one or more fluoroscope images to a computer; measuring one ormore angles related to the one or more fluoroscope images of themusculoskeletal system wherein the computer is configured to measure theone or more angles from the one or more fluoroscope images; displayingone or more metrics related to the one or more fluoroscope images on thedisplay of the computer; focusing the camera on a region of interest ofa patient wherein the computer illustrates a plurality of bones in theregion of interest on the display, wherein the computer identifies theplurality of bones from the one or more fluoroscope images, and whereinthe computer is configured to wait for a response from a surgical teamthat confirms the plurality of bones on the display are correctlyidentified by the computer; calculating one or more metrics generatedfrom the camera image data of the plurality of bones being manipulatedby a surgeon in real-time; displaying the one or more metrics from thecamera image data on the display wherein the display also illustratesmovement of the plurality of bones in real-time to support positioningof the plurality of bones using the one or more metrics from the cameraimage data to achieve a desired outcome and wherein the computer storesa position of the plurality of bones when the plurality of bones arepositioned to achieve the desired outcome.
 2. The method of claim 1further including the steps of: identifying vertebrae in the one or morefluoroscope images wherein the computer is configured to use computervision to recognize unique landmarks and features that distinguisheseach vertebra uniquely; displaying the computer identified vertebrae ona display of the computer; confirming that the identified vertebrae bycomputer vision on the display corresponds to the vertebrae on the oneor more fluoroscope images wherein computer vision is configured torecognize components of the musculoskeletal system despite variationsseen across a population wherein one of the surgical team reviews andprovides input to the computer that the computer vision has correctlyidentified the vertebrae; and preventing the computer from furthersurgical support if the identified vertebrae are not confirmed.
 3. Themethod of claim 2 further including the steps of: identifying endplatesof vertebrae in a fluoroscope image wherein the computer is configuredto use computer vision to recognize vertebra endplates; and measuring anangle between two vertebra endplates wherein the computer is configuredto measure the angle.
 4. The method of claim 3 further including a stepof measuring a sagittal Cobb angle, a coronal Cobb angle, or an axialrotation wherein the one or more fluoroscope images are used by thecomputer to calculate the metrics.
 5. The method of claim 2 furtherincluding the steps of: identifying pedicle screws in a fluoroscopeimage wherein the computer is configured to use computer vision torecognize pedicle screws; and measuring an angle between two pediclescrews identified by the computer wherein the computer is configured tomeasure the angle.
 6. The method of claim 2 further including the stepsof: identifying endplates of vertebrae in a fluoroscope image whereinthe computer is configured to use computer vision to recognize vertebraendplates identifying pedicle screws in the fluoroscope image whereinthe computer is configured to use computer vision to recognize pediclescrews; and measuring an angle between a pedicle screw and vertebraendplate wherein the computer is configured to measure the angle.
 7. Themethod of claim 2 further including the steps of: coupling at least onetarget to a vertebra; sending image data from the camera to the computerwherein the at least one target is in the field of view of the camera;measuring a position of the vertebra coupled to that at least one targetwherein the computer uses computer vision to determine a position of thevertebra from the position of the at least one target in real-time; anddisplaying a position of the vertebra in real-time on the display of thecomputer.
 8. The method of claim 7 further including the steps of:coupling pedicle screw extenders to pedicle screws on the spine whereinthe pedicle screw extenders are targets; measuring a spatialrelationship between the pedicle screw extenders using computer visionin real-time; relating positions of pedicle screw extenders to positionsof vertebrae wherein the computer calculates the positions of vertebrae;and displaying positions of vertebra relative to one another on thedisplay of the computer in real-time.
 9. The method of claim 1 furtherincluding the steps of: Measuring one or more metrics related to thedesired outcome using the image data and computer vision software as theplurality of bones are manipulated; and displaying the one or metricsrelated to the desired outcome on the display such that the surgeon canrapidly assimilate the information to adjust the plurality of bonestowards the desired outcome in real-time.
 10. The method of claim 9further including the steps of: comparing the one or metrics to apre-operative plan in real-time; storing the one or more metrics whenthe plurality of bones are positioned to achieve the desired outcome;and generating a structure to hold the plurality of bones in a positioncorresponding to the desired outcome from the stored one or moremetrics.
 11. The method of claim 1 wherein a joint is a spine andfurther including the steps of: taking a lateral and ananterior-posterior view of a spine region with the fluoroscope;measuring one or more angles related to a shape of the spine regionwherein the computer is configured to measure the one or more anglesfrom the one or more fluoroscope images; and displaying one or moremetrics related to the spine in a pre-surgical configuration.
 12. Themethod of claim 1 further including the steps of: coupling at least onetarget to a bone of a joint; sending image data from the camera to thecomputer wherein the at least one target is in the field of view of thecamera; measuring a position the bone of the joint coupled to that atleast one target wherein the computer uses computer vision to determinea position of the bone of the joint from the position of the at leastone target in real-time; and displaying a position of the bone of thejoint in real-time on a display of the computer.
 13. The method of claim1 further including the steps of: coupling a screw to each bone of ajoint wherein each screw is a target; measuring a spatial relationshipbetween the screws using computer vision in real-time; relatingpositions of each screw to positions of each bone of the joint whereinthe computer calculates the positions of the bones of the joint; anddisplaying positions of the bones of the joint relative to one anotheron the display of the computer in real-time.
 14. The method of claim 13further including the steps of: manipulating the joint; generatingmetrics related to the positions of the bones of the joint wherein thecomputer calculates the metrics; displaying the metrics related to theposition of the bones of the joint in real-time on the display of thecomputer; and storing real-time the metrics of a spine shape on thecomputer.
 15. A method of measuring a joint of the musculoskeletalsystem during surgery comprising: displaying an illustration of one ormore bones of the joint on a display coupled to a computer wherein thecomputer is configured to run computer vision software, wherein thecomputer vision software is configured to recognize components of themusculoskeletal system, wherein the computer vision software isconfigured to analyze and process digital images from a camera andextract high-dimensional data to make decisions similar to a humanvisual system, and wherein the one or more bones on the display areidentified on the display by the computer vision software; responding toa request by the computer to verify that the one or more bones of thejoint have been correctly identified on the display by the computervision software; coupling at least one target to a bone of the jointwherein the at least one target is in a field of view of the camera;sending image data from the camera to the computer in real-time;measuring a position of the bone coupled to the at least one targetwherein the computer uses the computer vision software to determine theposition of the bone from the position of the at least one target inreal-time; calculating one or more metrics generated from the image dataof the one or more bones of the joint being manipulated by a surgeon inreal-time; displaying the one or more metrics on the display wherein thedisplay also illustrates movement of the one or more bones in real-timeto support positioning of the one or more bones using the one or moremetrics to achieve a desired outcome and wherein the computer stores aposition of the plurality of bones when the plurality of bones arepositioned to achieve the desired outcome.
 16. The method of claim 15further including the steps of: manipulating a spine; generating spinemetrics related to positions of vertebrae wherein the computercalculates the spine metrics; displaying movement of images of the spinein real-time with the spine metrics on the display to supportpositioning of the vertebrae to a desired spine outcome; and storing thespine metrics of a spine shape that meets the desired spine outcome onthe computer.
 17. The method of claim 16 further including a step ofmodifying a rod shape to meet the spine outcome based on the storedreal-time metrics.
 18. The method of claim 17 further including thesteps of: displaying a sagittal Cobb angle in real-time as the spine ismanipulated; displaying a coronal Cobb angle in real-time as the spineis manipulated; displaying the axial rotation of the spine in real-timeas the spine is manipulated; and comparing the spine metrics to apre-operative metrics.
 19. The method of claim 15 further including thesteps of: coupling at least one target to a vertebra of a spine;measuring a position the vertebra coupled to that at least one targetwherein the computer uses computer vision to determine a position of thevertebra from the position of the at least one target coupled to thevertebra in real-time; and displaying a position of the vertebra inreal-time on the display of the computer.
 20. The method of claim 19further including the steps of: coupling pedicle screw extenders topedicle screws wherein the pedicle screw extenders are targets;measuring a spatial relationship between the pedicle screw extendersusing computer vision in real-time; relating positions of pedicle screwextenders to positions of vertebrae wherein the computer calculates thepositions of vertebrae; and displaying positions of vertebra relative toone another on the display of the computer in real-time.
 21. A method ofmeasuring the musculoskeletal system during surgery comprising the stepsof: taking a lateral and an anterior-posterior view of a spine regionwith the fluoroscope; sending fluoroscope images of the spine region toa computer; measuring one or more angles from the fluoroscope imagesusing the computer wherein the one or more angles measured from thefluoroscope images relate to a shape of the spine region; aiming thecamera at the spine region; displaying one or more metrics related tothe spine in a pre-surgical configuration on a display of the computerwherein the one or more metrics can include illustrations of the spineregion on a display of the computer in proximity to a surgical field ofan operating room wherein the computer is configured to run computervision software, wherein the computer vision software is configured torecognize components of the musculoskeletal system, wherein the computervision software is configured to analyze and process digital images fromthe camera and extract high-dimensional data to make decisions similarto a human visual system, and wherein each component within the spineregion on the display has been identified by the computer visionsoftware; responding to a request by the computer to verify that eachcomponent of the spine region have been correctly identified by thecomputer vision software to continue the computer support of thesurgery; coupling targets to vertebrae of the spine region wherein thetargets are in a field of view of a camera; sending image data of thespine region from the camera to a computer in real-time; measuringpositions of the vertebrae in the spine region wherein the computer usescomputer vision to determine a the positions of the vertebrae from thepositions of the targets in real-time; manipulating the spine region;calculating one or more metrics related to position of the vertebrae inthe spine region from the image data in real-time; displaying movementof images of the spine region in real-time with the one or more metricson the display to support positioning of the vertebrae to a desiredspine outcome wherein viewing the movement of the spine region with theone or more metrics allows the surgeon to rapidly assimilate sagittal,coronal, and rotational information; and storing the one or more metricsof the spine region when the spine region meets the desired spineoutcome on the computer.
 22. The method of claim 21 further includingthe steps of: identifying endplates of vertebrae in the fluoroscopeimages wherein the computer is configured to use computer vision torecognize vertebra endplates; and measuring an angle betweeninterpolated plane trajectories of two vertebra endplates wherein theangle is a Cobb angle.
 23. The method of claim 21 further including thesteps of: coupling pedicle screw extenders to pedicle screws in thespine region wherein the pedicle screw extenders are targets; measuringa spatial relationship between the pedicle screw extenders usingcomputer vision in real-time; relating positions of pedicle screwextenders to positions of vertebrae wherein the computer calculates thepositions of vertebrae; and displaying positions of vertebra relative toone another on a display of the computer in real-time.
 24. The method ofclaim 23 further including the steps of: displaying a sagittal Cobbangle in real-time as the spine region is manipulated; displaying acoronal Cobb angle in real-time as the spine region is manipulated;displaying the axial rotation of the spine in real-time as the spineregion is manipulated; and modifying a rod shape corresponding to theone or more metrics stored in the computer that meets the desired spineoutcome.